CRF Design for Clinical Studies: The Distinct Roles of Data Managers vs Biostatisticians
In medtech clinical trials, the Case Report Form (CRF) is more than a tool for collecting data—it’s the backbone of the study. From capturing critical safety outcomes to evaluating device performance, a well-designed CRF ensures that the study’s goals are met efficiently and reliably.
Achieving this balance requires input from two key roles: the data manager and the biostatistician. While their contributions may overlap in some areas, these roles serve distinct and complementary purposes. Understanding how these professionals work together can help medtech sponsors avoid common pitfalls in CRF design and maximise the success of their trials.
The Data Manager’s Role in CRF Design
Data managers are experts in the operational and technical aspects of CRF design. Their role is to ensure that data collection is standardised, consistent, and compliant with relevant guidelines.
Key responsibilities of a data manager include:
Formatting CRFs: Ensuring fields are user-friendly and compatible with electronic data capture (EDC) systems.
Regulatory Compliance: Aligning CRFs with industry standards such as CDASH (Clinical Data Acquisition Standards Harmonization).
Site Usability: Designing forms that facilitate accurate and consistent data entry across multiple trial sites.
For instance, a data manager might ensure that dropdown menus in the CRF prevent free-text responses, reducing the risk of inconsistencies. Their focus is on the practical and technical aspects of data collection.
The Biostatistician’s Role in CRF Design
Biostatisticians, on the other hand, approach CRF design from an analytical perspective. Their focus is on ensuring that the data collected aligns with the study’s endpoints and supports meaningful statistical analysis.
Key responsibilities of a biostatistician include:
Aligning Data with Study Objectives: Defining the variables that need to be captured to evaluate the endpoints outlined in the Statistical Analysis Plan (SAP).
Variable Definition: Ensuring that collected data supports statistical methods, such as properly coding categorical variables (e.g., mild/moderate/severe).
Derived Metrics: Identifying composite or derived variables that must be pre-defined in the CRF to support downstream analysis.
For example, in a post-market study evaluating a vascular device, the biostatistician would ensure that the CRF captures restenosis rates in a format that allows calculation of primary patency—a key endpoint. Their input ensures that no critical data points are overlooked.
Why Both Roles Are Essential to MedTech Trials
Although data managers and biostatisticians work towards the same goal—collecting high-quality data—their approaches and expertise are fundamentally different. Collaboration between these roles is essential for creating CRFs that are both operationally feasible and analytically robust.
1. Preventing Data Gaps
Without biostatistician oversight, CRFs may fail to capture key variables required for endpoint evaluation. For example:
In a stent study, a missing field for recording restenosis or target vessel occlusion could render the primary endpoint unanalysable.
Failure to specify timepoints for data collection (e.g., 12-month vs. 60-month follow-up) may result in incomplete datasets for secondary analyses.
2. Ensuring Data Compatibility
While data managers ensure that CRFs meet technical and regulatory standards, biostatisticians ensure the data is analyzable. Misalignment in variable formats can lead to delays or errors during analysis. For instance:
Categorical variables (e.g., adverse event severity) coded as free text at some sites and numeric values at others can complicate statistical programming.
3. Regulatory-Ready Analysis
In medtech trials, regulatory submissions rely heavily on robust statistical reporting. Biostatisticians ensure that CRFs are designed to collect all data necessary for generating high-quality, compliant analyses. For example:
Derived metrics like cumulative incidence rates must be pre-defined in the CRF to avoid regulatory scrutiny over post-hoc adjustments.
Misconceptions About CRF Design in MedTech
A common misconception in medtech trials is that data managers can fully handle CRF design. While data managers are essential for operationalising CRFs, their expertise does not extend to defining the analytical framework needed to support endpoints and hypotheses.
The Risks of Excluding Biostatistician Input
When biostatisticians are excluded from CRF design:
Key Variables May Be Missing: Critical fields for evaluating endpoints may be omitted.
Data May Be Misaligned: Improperly coded variables can lead to delays during analysis or errors in reporting.
Regulatory Challenges May Arise: Incomplete or improperly formatted data can result in regulatory delays or rejection.
By including biostatisticians early in the CRF design process, sponsors can avoid these risks and ensure their study remains on track.
Real-World Example: The Power of Collaboration
Consider a post-market surveillance study for a diagnostic device. The sponsor relies on CRFs to collect data on device performance across real-world clinical settings. Initially, the data manager designed the CRFs to focus on ease of use at the sites. However, the biostatistician identified a critical oversight: the CRFs did not include fields to track device calibration data, a key variable for assessing long-term performance trends. By collaborating, the data manager and biostatistician ensured the CRFs met both operational and analytical requirements, setting the stage for a successful regulatory submission.
Practical Steps for MedTech Sponsors
To ensure robust CRF design in your medtech trial, consider these steps:
Involve Biostatisticians Early: Engage your biostatistician during the CRF design phase to define variables and ensure alignment with study endpoints.
Foster Collaboration: Encourage close communication between data managers and biostatisticians to balance operational efficiency with analytical rigor.
Prioritise Regulatory Readiness: Design CRFs with regulatory requirements in mind to avoid costly delays during submission.
Final Thoughts
In medtech clinical trials, the success of your study depends on more than just collecting data—it depends on collecting the right data in the right way. Data managers and biostatisticians each bring unique expertise to CRF design, and their collaboration ensures that your trial is set up for operational efficiency, analytical validity, and regulatory success.
By recognising the complementary roles of these professionals, medtech sponsors can avoid common pitfalls and ensure their studies deliver meaningful, actionable results. If you’re planning a clinical trial and want to learn more about how to optimise CRF design, our team at Anatomise Biostats is here to help.
Consider the consequences if a medical doctor, without a formal medical education or licensing, were to diagnose and treat patients. Such a doctor might misunderstand symptoms, choose the wrong treatments, or even harm patients due to lack of understanding and experience. Similarly, an unqualified biostatistician might incorrectly analyse data, misinterpret statistical significance, or fail to recognise biases and patterns essential for accurate conclusions. These errors, when compounded across studies and publications, create a domino effect, misleading the medical community and affecting clinical guidelines that doctors worldwide follow.
When biostatistics work is flawed due to lack of proper training, the evidence that supports clinical decision-making is compromised. The gravity of these potential errors is amplified because biostatistics underpins clinical trial outcomes, which are often used to secure regulatory approval and define the standards for how to treat diseases. If flawed analysis leads to approving ineffective or harmful treatments, patients could suffer adverse effects from what they believe are safe therapies. In this sense, an unqualified biostatistician is even more dangerous than an unlicensed doctor, as their errors can influence the treatment decisions of countless doctors, each one putting their patients at risk based on incorrect or incomplete data.
Without proper qualifications, a biostatistician’s work can lead to harmful outcomes. This is because the analysis they perform underpins the scientific evidence that doctors rely on to make clinical decisions and guide patient care.
Why a Coursework Masters of Biostatistics an indispensable foundation
High-quality biostatistics programs offer advanced, in-depth training that goes far beyond basic statistical application. One of the core skills instilled is the ability to identify gaps in knowledge and continually adapt to the specific demands of each unique clinical trial. A competent biostatistician isn’t just someone who knows how to apply a set of methods; they are a problem-solver equipped to navigate complex, evolving situations, often needing to research, adapt, or even develop new techniques as each clinical context requires.
Unlike a research-based master’s thesis, which typically hones expertise in a narrow area, a coursework master’s in biostatistics emphasises a broad, structured understanding of the field, preparing individuals to apply statistical techniques accurately in a clinical context. Rigorous training in biostatistics is essential because the stakes are high, and the work of a biostatistician directly influences the treatment approaches trusted by healthcare providers around the world.
A hallmark of a quality biostatistics program is it’s focus on cultivating a mindset of critical evaluation and adaptability. Rather than simply learning a fixed set of methods, students are taught to understand the foundational principles of statistics and how to apply them thoughtfully across different clinical scenarios. This training includes learning how to question assumptions, test the validity of models, and assess the appropriateness of methods in light of each study’s design and data characteristics. It also involves learning how to identify situations where the standard, previously used methods may not suffice—an ability that can only come from a deep understanding of the mathematical principles underpinning statistical techniques.
The mathematical underpinnings of statistical tests can be subtle and intricate. Without specialised training, there’s a high risk that these mathematical nuances will be overlooked or mishandled. For example, failure to correctly adjust for confounding variables can make it appear as though a treatment effect exists when it doesn’t, leading to erroneous conclusions that could harm patients if implemented in clinical practice.
A well-prepared biostatistician is not only familiar with a wide range of statistical tools but also understands when each tool is appropriate, and more importantly, when it may be insufficient. Clinical trials often present unique challenges, such as complex interactions between variables, confounding factors, and datasets that may not conform neatly to traditional statistical models. In these cases, biostatisticians trained to think critically and independently can recognise that the standard approaches may fall short and are capable of researching novel methods, exploring the latest advancements, and adapting techniques to better fit the data at hand. This ability to assess, research, and innovate rather than rigidly apply textbook methods is what makes a biostatistician invaluable to clinical research.
Advanced biostatistics programs emphasise this flexibility, often incorporating coursework in emerging statistical methods, machine learning, and adaptive designs that are becoming increasingly relevant in modern clinical trials. These programs also provide hands-on training with real-world data, equipping students to handle the messy, imperfect datasets typical in clinical research. Graduates from rigorous programs gain the skills needed to work with a high degree of precision, recognising the limitations of each approach and adapting their methods to provide the most reliable analysis possible.
This commitment to continuous learning and adaptability is essential, particularly in a field as fast-evolving as clinical biostatistics. New statistical models, computational methods, and technologies are constantly emerging, offering powerful new ways to analyse data and uncover insights that would be missed with conventional methods. Biostatisticians trained to think critically and assess what they do not yet know are equipped to stay at the forefront of these advancements, ensuring that clinical trial data is analysed with the most effective and current techniques.
Individuals without this specialised training or with training from adjacent fields may lack this advanced skill set. While they may be familiar with statistical software and certain techniques, they often lack the deeper statistical grounding that allows them to identify gaps in their own knowledge, research novel techniques, and apply methods flexibly. They may rely more heavily on familiar, pre-existing methods, even when these approaches are suboptimal for the specific demands of a new clinical trial.
In clinical research, it’s critical to distinguish between fields that may seem related to biostatistics but lack the specialised training needed for rigorous clinical trial analysis. Adjacent disciplines such as biomedical engineering or bioinformatics, while valuable in their own right, do not provide the depth and specificity of statistical training required for high-stakes clinical biostatistics. Clinical trials demand a comprehensive understanding of advanced statistical methods, hypothesis testing, probability theory, and the practical challenges inherent in real-world clinical data. Without this foundation, there is a high risk that even a highly skilled professional in an adjacent field may misinterpret trial data or apply suboptimal models, potentially jeopardising trial results.
While adjacent fields like biomedical engineering and bioinformatics serve as valuable components to clinical research teams, they do not replace a biostatistician in terms of the depth of statistical expertise required to conduct clinical trials safely and effectively. Additionally, even within biostatistics itself, the rigour and quality of training can vary widely between institutions. A high-quality biostatistics qualification, grounded in coursework and practical experience, is essential to ensure that biostatisticians are fully prepared to meet the demands of clinical trial analysis, providing reliable evidence that healthcare providers can depend on to guide safe, effective patient care.
Core statistical concepts: Beyond Basic Stats
When we think about clinical trials, we often picture doctors, patients, and maybe lab scientists—but behind every trial is a biostatistician. They’re responsible for interpreting the data in a way that uncovers whether a treatment truly works, and just as importantly, whether it’s safe. On the surface, this might sound like standard statistics, but the reality is far more complex. Clinical trials involve intricate designs, variable data, and outcomes that hinge on precisely the right analytical approach. Here’s why a biostatistician needs a Master’s degree in biostatistics to navigate this terrain.
The Power Calculation: Not Just Plugging in Numbers
One of the most fundamental tasks in clinical trials is calculating statistical power—essentially, determining the sample size required to detect a treatment effect if it exists. While it might sound as simple as choosing a sample size, calculating power is actually a multi-layered process, filled with nuances that require advanced training.
A biostatistician needs to understand how effect size, variability, sample size, and study design all interact. For instance, they can’t simply use a pre-set formula; they must examine assumptions about the patient population, factor in dropout rates, and sometimes even simulate different scenarios to see how robust their sample size calculation is. If the sample size is too small, the study could miss a true treatment effect, leading to the incorrect conclusion that a treatment is ineffective. Too large, and it wastes resources and could expose patients to unnecessary risk.
An advanced biostatistics program should explore how to conduct sensitivity analyses, interpret simulation results, and understand the trade-offs in different power calculation approaches. These skills can be impractical to cultivate on the job without a solid foundation.
Hypothesis Testing: Far More Than Just a P-value
Hypothesis testing often gets reduced to p-values, but in clinical trials, p-values are just the tip of the iceberg. Deciding how to structure a hypothesis test is a skill that requires an in-depth understanding of the trial design, data type, and statistical limitations. P-values themselves are affected by factors like sample size and effect size, and they depend on correct assumptions about the data. If these assumptions are even slightly off, the results could be misleading. Additionally, a significant p value is not necessarily clinically meaningful – an effect size must be carefully considered.
Suppose a trial includes multiple subgroups, such as different age ranges, where treatment response might vary. A biostatistician needs to decide whether to test each group separately or combine them, taking into account the risk of inflating the false positive rate. They may have to employ adjustments like the Bonferroni correction or false discovery rate, each with its own implications for the results’ reliability. Knowing when and how to apply these adjustments requires expertise in statistical trade-offs—a skill set that goes beyond basic training.
Bayesian Modelling: The Complexity of Integrating Prior Information
In clinical trials, Bayesian modelling offers the flexibility to incorporate prior information, which can be crucial when there’s existing data on similar treatments. But building a Bayesian model is not as simple as adding a prior and letting the data “speak.” Bayesian analysis is an iterative, highly contextual process that involves understanding the nuances of prior selection, data updates, and model convergence.
For example, in a trial with limited data, the biostatistician might consider a prior based on past studies. But they need to ensure that the prior doesn’t overpower the current data, especially if the populations differ in meaningful ways. They’ll also have to assess how sensitive the model is to the chosen prior—small changes can have a large impact on the results. Once the model is built, they will test it with simulations, iteratively refine their approach, and apply computational techniques like Markov Chain Monte Carlo methods to ensure accurate estimates.
Core skills include how to choose and validate priors, handle computational challenges, and interpret Bayesian results in a way that is both statistically valid and clinically meaningful. Without this background, Bayesian methods could be misapplied, leading to conclusions that are overly dependent on prior data, potentially skewing the trial’s findings.
Handling Confounding Variables: Getting to the True Treatment Effect
Confounding variables are one of the most significant challenges in clinical trials. These are external factors that could influence both the treatment and the outcome, creating a false impression of effect. Managing confounding variables isn’t as simple as throwing all variables into a model. It involves selecting the right approach—whether that’s stratification, regression adjustment, or propensity score matching—to isolate the treatment’s actual impact.
Imagine a trial assessing the effect of a heart medication where younger patients tend to recover faster. If age isn’t properly accounted for, the results might suggest that the treatment is effective, simply because younger patients are overrepresented in the treatment group. Handling such confounding factors involves understanding the dependencies between variables, testing assumptions, and assessing the adequacy of different adjustment techniques.
Biostatistics programs address these complexities, teaching biostatisticians how to identify and handle confounders, use advanced models like inverse probability weighting, and validate their adjustments with sensitivity analyses. This is not something that can be mastered without a solid foundation in statistics and it’s application to medicine.
A practical example:
Consider a clinical trial evaluating an innovative cardiac monitoring device intended to reduce adverse cardiovascular events in a diverse patient population, with participants spanning a wide range of ages, co-morbidities, and cardiovascular risk profiles. The complexity of this study lies not only in the heterogeneity of the patient population but also in the need to accurately capture the device’s effectiveness over extended time periods and in varied real-world contexts. Here, standard statistical methods may fail to capture the full picture; without careful investigation and adaptation, these methods could miss critical variations in device effectiveness across different patient subgroups. Missteps in analysis could lead to misguided conclusions, resulting in the misapplication of the device or failure to recognise its specific benefits for certain populations.
An unqualified biostatistician, seeing only the broad structure of the trial, might select standard statistical approaches such as repeated measures analysis or proportional hazards models, assuming that the device’s impact can be summarised uniformly across patients and time. These methods, while effective in certain contexts, may oversimplify the true complexity of the data. For instance, these approaches may overlook significant patient-specific variations, assuming all patients respond similarly over time, and fail to address potential dependencies across repeated measurements. In doing so, they risk obscuring insights into how the device performs across age groups, co-morbidity profiles, or geographic regions.
A competent biostatistician, however, would recognise that such a complex, dynamic scenario demands a more tailored and investigative approach. They would start by reviewing trial specifics—population diversity, data structure, and endpoints—and identifying the particular challenges these present. This initial assessment might lead them to consider a range of advanced modelling techniques, from hierarchical models and frailty models to time-varying covariate models, evaluating each option to find the best fit for the study’s unique demands.
For instance, a hierarchical model could capture variability at multiple levels—such as individual patients, treatment centres, or geographic clusters—allowing the biostatistician to account for factors that might cluster within sites or subgroups. If, for example, patients from one geographic area tend to experience more adverse events, a hierarchical model would help isolate these effects, ensuring they don’t skew the treatment outcomes. A frailty model, on the other hand, might be more appropriate if there are unobserved variables influencing patient outcomes, such as genetic predispositions or lifestyle factors that impact how individuals respond to the device. Each model offers benefits but comes with specific assumptions and limitations, requiring the biostatistician to weigh these factors carefully.
The biostatistician would then move beyond selecting a method, entering a phase of critical evaluation and testing. They perform model diagnostics to check assumptions, such as independence and proportional hazards, assessing how well each model fits the trial data. If they find that patient characteristics change over time, influencing treatment response, they may pivot toward a time-varying covariate model. Such a model could capture how the effectiveness of the device changes with patient health fluctuations, an essential insight in trials where health status is dynamic. Rather than assuming proportional effects across time, this approach would allow the analysis to reflect real-world shifts in patient health and co-morbidity, enhancing the relevance of the results for long-term patient care.
In addition, the biostatistician may implement advanced stratification techniques or subgroup analyses, aiming to parse out the effects of specific co-morbidities like diabetes or chronic kidney disease. These approaches are not simply a matter of segmenting data; they require careful control of confounding variables and an understanding of how stratification affects power and interpretation. The biostatistician could explore techniques such as propensity score weighting or covariate balancing to create comparable subgroups, helping to isolate the device’s effect on each subgroup with minimal bias. This ensures that the treatment effect estimation is not conflated with unrelated patient characteristics, like age or pre-existing health conditions, which could distort the true efficacy of the device.
Because of the trial’s longitudinal design, the biostatistician would also need to research and carefully apply methods that accommodate time-dependent covariates. They might examine the appropriateness of flexible parametric survival models over the traditional Cox model, especially if patient health or response to treatment fluctuates significantly over time. By reviewing the latest literature and comparing models through simulation studies, the biostatistician can determine which methods best capture the time-varying nature of the data without introducing artefacts or biases. For instance, a flexible model might reveal periods during which the device is particularly effective, or it could show diminishing efficacy as patients’ health profiles evolve, offering critical insights into when and for whom the device provides the most benefit.
In this rigorous process, the biostatistician doesn’t simply apply methods—they conduct an iterative investigation, refining their approach with each step. Sensitivity analyses, for example, might be run to determine how robust findings are to different modelling choices or to evaluate the impact of unmeasured confounders. Through this iterative process, they test assumptions, explore the validity of each approach, and adjust techniques to ensure that their final analysis captures the device’s effectiveness in a nuanced, clinically relevant way. This stands in contrast to a one-size-fits-all analysis, where insights into key variations across patient subgroups may be lost.
Ultimately, the advanced approach adopted by a qualified biostatistician goes beyond statistical rigour—it provides a comprehensive, meaningful picture of the device’s real-world effectiveness. By thoroughly investigating and validating each method, the biostatistician ensures that their analysis accurately reflects how the device performs across diverse patient populations. This depth of analysis provides doctors with reliable, specific insights into which patients are most likely to benefit, supporting safer, more personalised treatment decisions in real-world clinical settings.
Why Outsourcing a Biostatistics Team is Pivotal to the Success of your Clinical Trial
Clinical trials are among the most critical phases in bringing a medical device or pharmaceutical product to market, and ensuring the accuracy and integrity of the data generated is essential for success. While some companies may feel confident relying on their internal teams, especially if they have expertise in AI or data science, managing the full scope of biometrics in clinical trials often requires far more specialised skills. Building a dedicated in-house team may seem like a natural next step, but it can involve significant time, cost, and resource investment that can sometimes be underestimated.
Outsourcing biometrics services offers a streamlined, cost-effective alternative, providing access to a team of specialists in statistical programming, quality control, and regulatory compliance. Much like outsourcing marketing or legal services, entrusting biometrics to an external team allows businesses to focus on their core strengths while ensuring the highest standards of data accuracy and regulatory alignment. In this article, we explore why outsourcing biometrics is a smarter approach for clinical trials, offering the expertise, flexibility, and scalability needed to succeed.
1. Expertise Across Multiple Disciplines
Clinical trials require a blend of specialised skills, from statistical programming and data management to quality control and regulatory compliance. Managing these diverse requirements internally can stretch resources and may lead to oversights. When outsourcing to a biometrics team, companies can access a broad range of expertise across all these critical areas, ensuring that every aspect of the trial is handled by specialists in their respective fields.
Instead of spreading resources thin across a small internal team, outsourcing offers a more efficient approach where every key area is covered by experts, ultimately reducing the risk of errors and enhancing the quality of the trial data.
2. Avoid Bottlenecks and Delays
Managing the data needs of a clinical trial requires careful coordination, and internal teams can sometimes face bottlenecks due to workload or resource limitations. Unexpected delays, such as staff absences or project overload, can slow progress and increase the risk of missed deadlines.
Outsourcing provides built-in flexibility, where a larger, more experienced team can step in when needed, ensuring work continues without interruption. This kind of seamless handover keeps the trial on track and avoids the costly delays that might arise from trying to juggle too many responsibilities in-house.
3. Improved Data Quality Through Redundancy
One of the advantages of outsourcing biometrics is the added level of redundancy it offers. In-house teams, particularly small ones, may not have the capacity for thorough internal quality checks, potentially allowing errors to slip through.
Outsourced teams typically have multiple layers of review built into their processes. This ensures that data undergoes several levels of scrutiny, significantly reducing the risk of unnoticed mistakes and increasing the overall reliability of the analysis.
4. Flexibility and Scalability
The nature of clinical trials often shifts, with new sites, additional data points, or evolving regulatory requirements. This creates a demand for scalability in managing the trial’s data. Internal teams can struggle to keep up as the project grows, sometimes leading to bottlenecks or rushed work that compromises quality.
Outsourcing biometrics allows companies to adapt to the changing scope of a trial easily. A specialised team can quickly scale its operations to handle additional workload without compromising the timeline or quality of the analysis.
5. Ensuring Regulatory Compliance
Meeting regulatory requirements is a critical aspect of any clinical trial. From meticulous data documentation to adherence to best practices, there are stringent standards that must be followed to gain approval from bodies like the FDA or EMA.
Outsourcing to an experienced biometrics team ensures that these standards are met consistently. Having worked across multiple trials, outsourced teams are well-versed in the latest regulations and can ensure that all aspects of the trial meet the necessary compliance requirements. This reduces the risk of costly rejections or trial delays caused by non-compliance.
6. Enhanced Data Security and Infrastructure
Handling sensitive clinical trial data requires secure systems and advanced infrastructure, which can be costly for companies to manage internally. Maintaining this infrastructure, along with the necessary cybersecurity measures, can quickly escalate expenses, especially for smaller in-house teams.
By outsourcing biometrics, companies gain access to teams with pre-existing secure infrastructure designed specifically for clinical data. This not only reduces costs but also mitigates the risk of data breaches, ensuring compliance with privacy regulations like GDPR.
7. Hidden Challenges of Building an In-House Team
While building an in-house biometrics team might seem appealing, it comes with it’s hidden challenges and costs that are easily overlooked. Recruitment, training, administrative load and retention all contribute to a growing budget, along with HR costs and the ongoing need to invest in tools and advanced infrastructure to keep the team effective.
Outsourcing offers a clear financial benefit here. Companies can bypass many resource draining activities and gain immediate access to a team of experts, without having to worry about ongoing staff management or the investment in specialised tools.
8. Unbiased Expertise
Internal teams may face pressure to align with existing company practices or preferences, which can sometimes lead to biased decisions when it comes to methodology or quality control. Outsourced teams are entirely independent and focused solely on delivering objective, high-quality results. This ensures that the best statistical methods are applied, without the potential for internal pressures to sway critical decisions.
The Case for Outsourcing Biometrics
Clinical trials are complex and require a range of specialised skills to ensure their success. While building an in-house team might seem like an intuitive solution, it often introduces unnecessary risks, hidden costs, and logistical challenges. Outsourcing biometrics to a specialised team offers a streamlined, scalable solution that ensures trial data is handled with precision and integrity, while maintaining regulatory compliance.
By leveraging the expertise of an external biometrics team, companies can focus on their core strengths—whether it’s developing a breakthrough medical device or innovating in their field—while leaving the complexities of biometrics to the experts.
If you’re preparing for your next clinical trial and want to ensure reliable, accurate, and compliant results, contact Anatomise Biostats today. Our expert biometrics team is ready to support your project and deliver the results you need to bring your medical device to market with confidence.
The ethical and accurate handling of data is paramount in the domain of clinical research. As the demand for data-driven clinical insights continues to grow, researchers face challenges in balancing the need for accuracy with the availability of data and the imperative to protect sensitive information. In situations where quality real patient data is not available, synthetic data can be the most reliable data source from which to derive predictive insights. Synthetic data can be more cost-effective and time-efficient in many cases than acquiring the equivalent real data.
Synthetic data must be differentiated from fake data. In recent years there has been much controversy concerning fake data detected in published journal articles which have previously passed peer review, particularly in an academic context. As one study is generally built upon assumptions formed by the results of another, this preponderance of fake data has really had a catastrophic impact on our ability to trust any published scientific research, regardless of whether the study at hand also contains fake data. It has become clear that the implementation of increased quality control standards for all published research needs to be prioritised.
While synthetic data is not without it’s own pitfalls, the key difference between synthetic and fake data lies in it’s purpose and authenticity. Synthetic data is designed to emulate real-world data for specific use cases, maintaining statistical properties without revealing actual (individual) information. On the other hand, fake data is typically fabricated and may not adhere to any real-world patterns or statistics.
In clinical research, the use of real patient data is fraught with privacy concerns and other ethical considerations. Accurate and consistent patient data can also be hard to come by for other reasons such as heterogeneous recording methods or insufficient disease populations. Synthetic data is emerging as a powerful solution to navigate these limitations. While accurate synthetic data is not a trivial thing to generate, researchers can harness advanced algorithms and models built by expert data scientists to generate synthetic datasets that faithfully mimic the statistical properties and patterns of real-world patient and other data. This allows researchers to simulate and predict relevant clinical outcomes in situations where real data is not readily available, and do so without compromising individual patient privacy.
A large proportion of machine learning models in an AI context are currently being trained on synthetic rather than real data. This is largely because using generative models to create synthetic data tends to be much faster and cheaper than collecting real-world-data. Real-world data can at times lack sufficient diversity to make insights and predictions truly generalisable.
Both the irresponsible use of synthetic data and the generation and application of fake data in academic, industry and clinical research settings can have severe consequences. Whether stemming from dishonesty or incompetence, the misuse of fake data or inaccurate synthetic data poses a threat to the integrity of scientific inquiry.
This following sections define and delineate between synthetic and fake data as well as summarise the key applications of synthetic data in clinical research as compared to the potential pitfalls associated with the unethical use of fake data.
Synthetic Data:
Synthetic data refers to artificially generated data that mimics the statistical properties and patterns of real-world data. It is created using various algorithms, models, or simulations to resemble authentic patient data as closely as possible. It may do so without containing any real-world identifying information about individual patients comprising the original patient sample from which it was derived.
Synthetic data can be used in situations where privacy, security, or confidentiality concerns make it challenging to access or use real patient data. It can also be used in cases where an insufficient volume of quality patient data is available or where existing data is too heterogeneous to draw accurate inferences, such as is typically the case with rare diseases. It can potentially be employed in product testing to create realistic scenarios without subjecting real patients to unnecessary risk.
3 key use cases for synthetic data in clinical research
1. Privacy Preservation:
– Synthetic data allows researchers to conduct analyses and develop statistical models without exposing sensitive patient information. This is particularly crucial in the healthcare and clinical research sectors, where maintaining patient confidentiality is a legal and ethical imperative.
2. Robust Testing Environments:
– Clinical trials and other experiments related to product testing or behavioural interventions often necessitate testing in various scenarios. Synthetic data provides a versatile and secure testing ground, enabling researchers to validate algorithms and methodologies without putting real patients at risk.
3. Data Augmentation for Limited Datasets:
– In situations where obtaining a large and diverse dataset is challenging, synthetic data serves as a valuable tool for augmenting existing datasets. This aids in the development of more robust models and generalisable findings. A data set can be made up of varying proportions of synthetic versus real-world data. For example, a real world data set may be fairly large but lack diversity on the one hand, or small and overly heterogeneous on the other. The methods of generating synthetic data to augment these respective data sets would differ in each case.
Fake Data:
Fake data typically refers to data that is intentionally fabricated or inaccurate due to improper data handling techniques. In situations of misuse it is usually combined with real study data to give misleading results.
Fake data can be used ethically for various purposes, such as placeholder values in a database during development, creating fictional scenarios for training or educational purposes, or generating data for scenarios where realism is not crucial. Unfortunately in the majority of notable academic and clinical cases it has been used with the deliberate intention to mislead by doctoring study results and thus poses a serious threat to the scientific community and the general public.
.There are three key concerns with fake data.
1. Academic Dishonesty:
– Some researchers may be tempted to fabricate data to support preconceived conclusions, meet publication deadlines or attain competitive research grants. After many high profile cases in recent years it has become apparent that this is a pervasive issue across academic and clinical research. This form of academic dishonesty undermines the foundation of scholarly pursuits and erodes the trust placed in research findings.
2. Mishaps and Ineptitude:
– Inexperienced researchers may inadvertently create fake data, whether due to poor data collection practices, computational errors, or other mishaps. This unintentional misuse can lead to inaccurate results, potentially rendering an entire body of research unreliable if it remains undetected.
3. Erosion of Trust and Reproducibility:
– The use of fake data contributes to the reproducibility crisis in scientific research. One study found that 70% of studies cannot be reproduced due to insufficient reporting of data and methods. When results cannot be independently verified, trust in the scientific process diminishes, hindering the advancement of knowledge. The addition of fake data into this scenario makes replication and thus verification of study results all the more challenging.
In an evolving clinical research landscape, the responsible and ethical use of data is paramount. Synthetic data stands out as a valuable tool in protecting privacy, advancing research, and addressing the challenges posed by sensitive information – assuming it is generated as accurately and responsibly as possible. On the other hand, the misuse of fake data undermines the integrity of scientific research, eroding trust and impeding the progress of knowledge and it’s real-world applications. It is important to stay vigilant against bias in data and employ stringent quality control in all data contexts of data handling.
P values are so ubiquitous in clinical research that it’s easy to take for granted that they are being understood and interpreted correctly. After-all, one might say, they are just simple proportions and it’s not brain surgery. At times, however, its’ the simplest of things that are easiest to overlook. In fact, the definitions and interpretations of p values are sufficiently subtle that even a minute pivot from an exact definition can lead to interpretations that are wildly misleading.
In the case of clinical trials, p values have a momentous impact on decision making in terms of whether or not to pursue and invest further into the development and marketing of a given therapeutic. In the context of clinical practice p values drive treatment decisions for patients as they essentially comprise the foundational evidence upon which these treatment decisions are made. This is perhaps as it should be, as long as the definition of p values and their interpretations are sound.
A counter-point to this is the bias towards publishing only studies with a statistically significant p value, as well as the fact that many studies are not sufficiently reproducible or reproduced. This leaves clinicians with an impression that evidence for a given treatment is stronger than the full picture would suggest. This however is a publishing issue rather than an issue of significance tests themselves. This article focusses on interpretation issues only.
As p values apply to the interpretation of both parametric and non-parametric tests in much the same way, this article will focus on parametric examples.
Interpreting p values in superiority/difference study designs
This refers to studies where we are seeking to find a difference between two treatment groups or between a single group measured at two time points. In this case the null hypothesis is that there is no difference between the two treatment groups or no effect of the treatment, as the case may be.
According to the significance testing framework all p values are calculated based upon an assumption that the null hypothesis is true. If a study yields a p value of 0.05, this means that we would expect to see a difference between the two groups at least as extreme as the observed effect 5% of the time; if the study were to be repeated. In other words, if there is no true difference between the two treatment groups and we ran the experiment 20 times on 20 independent samples from the same population, we would expect to see a result this extreme once out of the 20 times.
This of course is not a very helpful way of looking at things if our goal is to make a statement about treatment effectiveness. The inverse likely makes more intuitive sense: if were were to run this study 20 times on distinct patient samples from the same population, 19 out of 20 times we would not expect a result this extreme if there was no true effect. Based on the rarity of the observed effect, we conclude that likelihood of the null hypothesis being the optimal explanation of the data is sufficiently low that we can reject it. Thus our alternative research hypothesis, that there is a difference between the two treatments, is likely to be true. As the p value does not tell us whether the difference is a positive or negative direction, care should of course be taken to confirm which of the treatments holds the advantage.
P values in non-inferiority or equivalence studies.
In non-inferiority and equivalence studies a non-statistically significant p value can be a welcome result, as can a very low p value where the differences were not clinically significant, or where the new treatment is shown to be superior to the standard treatment. By only requiring the treatment not to be inferior, more power is retained and a smaller sample size can be used.
The interpretation of the p value is much the same as for superiority studies, however the implications are different. In these types of studies it is ideal for the confidence intervals for the individual treatment effects to be narrow as this provides certainty that the estimates obtained are accurate in the absence of a statistically significant difference between the two estimates.
While alternatives to p values exist, such as Bayesian statistics, these statistics have limitations of their own and are subject to the same propensity for misuse and misinterpretation as frequentist statistics are. Thus it remains important to take caution in interpreting all statistical results.
What p values do not tell you
A p value of 0.05 is not the same as saying that there is only a 5% chance that the treatment wont work. Whether or not the treatment works in the individual is another probability entirely. It is also not the same as saying there is a 5% chance of the null hypothesis being true. The p value is a statistic that is based on the assumption that the null hypothesis is true and on that basis gives the likelihood of the observed result.
Nor does the p value represent the chance of making a type 1 error. As each repetition of the same experiment produces a different p value, it does not make sense to characterise the p value as the chance of incorrectly rejecting the null hypothesis ie making a type one error. Instead, an alpha cut-off point of 0.05 should be seen as indicating a result rare enough under the null hypothesis that we are now willing to reject the null as the most likely explanation given the data. Under a type-one error alpha of 0.05 this decision is expected to be wrong 5% of the time, regardless of the p value achieved in the statistical test. The relationship between the critical alpha and statistical power is illustrated below.
Another misconception is that a small p value provides strong support for a given research hypothesis. In reality a small p value does not necessarily translate to a big effect, nor a clinically meaningful one. The p value indicates a statistically significant result, however it says nothing about the magnitude of the effect or whether this result is clinically meaningful in the context of the study. A p value of 0.00001 may appear to be a very satisfactory result, however if the difference observed between the two groups is very small then this is not always the case. All it would be saying is that “we are really really sure that there is only minimal difference between the two treatments”, which in a superiority design may not be as desired.
Minimally important difference (MID)
This is where the importance of pre-defining a minimally important difference (MID) becomes evident. The MID, or clinically meaningful difference. should be defined and quantified in the design stage before the study is to be undertaken. In the case of clinical studies this should generally be done in consultation with the clinician or disease expert concerned.
The MID may take different forms depending on whether a study is a superiority design, versus an equivalence or non-inferiority design. In the case of a superiority design or where the goal of the study is to detect a difference, the MID is the threshold of minimum difference at which we would be willing to consider the new treatment worth pursuing over the standard treatment or control being used as the comparator. In the case of a non-inferiority design the MID would be the minimum lower threshold at which we would still consider the new treatment as equally effective or useful as the standard treatment. Equivalence design on the other hand may sometimes rely on an interval around the standard treatment effect.
When interpreting results of clinical studies it is of primary importance to keep a clinically meaningful difference in mind, rather than defaulting to the p value in isolation. In cases where the p value is statistically significant, it is important to ask whether the difference between comparison groups is also as large as the MID or larger.
Confidence Intervals
All statistical tests that involve p values can produce a corresponding confidence interval for the estimates. Unlike p values, confidence intervals do not rely on an assumption of the null hypothesis but rather on the assumption that the sample approximates the population of interest. A common estimate in clinical trials where confidence intervals become important is the treatment effect. Very often this translates to the difference in means of a surrogate endpoint between two groups, however confidence intervals are also important to consider for individual group means/ treatment effects, which are an estimate of the population means of the endpoint in these distinct groups/treatment categories.
Confidence interval for the mean
A 95% confidence interval of the estimate of the mean indicates that, if this study were to be repeated, the mean value is expected to fall within this interval 95% of the time. While this estimate is based on the real mean of the study sample our interest remains in making inferences about the wider population who might later be subject to this treatment. Thus inferentially the observed mean and it’s confidence interval are both considered an estimate of the population values.
In a nutshell the confidence interval indicates how sure we can be of the accuracy of the estimate. A narrower interval indicates greater confidence and a wider interval less. The p value of the estimate indicates how certain we can be of this result, ie the interval itself.
Confidence interval for the mean difference, treatment effects or difference in treatment effects
The mean difference in treatment effect between two groups is an important estimate in any comparison study, from superiority to non-inferiority clinical trial designs. Treatment response is mainly ascertained from repeated measures of surrogate endpoint data on the individual level. One form of mean difference is repeated measures data from the same individuals at different time points, these individuals’ differences could then be compared between two independent treatment groups. In the context of clinical trials, confidence intervals of the mean difference can relate to an individual’s treatment effect or to group differences in treatment effects.
A 95% Confidence interval of the mean difference in treatment effect indicates that 95 per cent of the time, if this study were to be repeated, the true difference in treatment effect between the groups is expected to fall within this interval. A confidence interval containing zero indicates that a statistically significant difference between the two groups has not been found. Namely, if part of the time the true population value representing the difference is expected to fall above zero on the number line and part of the time to fall below zero, indicating a difference in the opposite direction, we cannot be sure whether one group is higher or lower than the other.
Much ho-hum has been made of p values in recent years but they are here to stay. While alternatives to p values exist, such as Bayesian methods, these statistics have limitations of their own and are subject to the same propensity for misuse and misinterpretation as frequentist statistics are. Thus it remains important to take caution in interpreting all statistical results.
Sources and further reading:
Gao, P-Values – A chronic conundrum, BMC Medical Research Methodology (2020), 20:167 https://doi.org/10.1186/s12874-020-01051-6
The Royal College of Ophthalmologists, The clinician’s guide to p values, confidence intervals, and magnitude of effects, Eye (2022) 36:341–342; https://doi.org/10.1038/s41433-021-01863-w
As a medical researcher or a small enterprise in the life sciences industry, you are likely to encounter many experts using statistical and computational techniques to study biological, clinical and other health data. These experts can come from a variety of fields such as biostatistics, bioinformatics, biometrics, clinical data science and epidemiology. Although these fields do overlap in certain ways they differ in purpose, focus, and application. All four areas listed above focus on analysing and interpreting either biological, clinical data or public health data but they typically do so in different ways and with different goals in mind. Understanding these differences can help you choose the most appropriate specialists for your research project and get the most out of their expertise. This article will begin with a brief description of these disciplines for the sake of disambiguation, then focus on biostatistics and bioinformatics, with a particular overview of the roles of biostatisticians and bioinformatics scientists in clinical trials.
Biostatisticians
Biostatisticians use advanced biostatistical methods to design and analyse pre-clinical experiments, clinical trials, and observational studies predominantly in the medical and health sciences. They can also work in ecological or biological fields which will not be the focus of this article. Biostatisticians tend to work on varied data sets, including a combination of medical, public health and genetic data in the context of clinical studies. Biostatisticians are involved in every stage of a research project, from planning and designing the study, to collecting and analysing the data, to interpreting and communicating the results. They may also be involved in developing new statistical methods and software tools. In the UK the term “medical statistician” has been in common use over the past 40 years to describe a biostatistician, particularly one working in clinical trials, but it is becoming less used due to the global nature of the life sciences industry.
Bioinformaticians
Bioinformaticians use computational and statistical techniques to analyse and interpret large datasets in the life sciences. They often work with multi-omics data such as genomics, proteomics transcriptomics data and use tools such as large databases, algorithms, and specialised software programs to analyse and make sense of sequencing and other data. Bioinformaticians develop analysis pipelines and fine-tune methods and tools for analysing biological data to fit the evolving needs of researchers.
Clinical data scientists
Data scientists use statistical and computational modelling methods to make predictions and extract insights from a wide range of data. Often, data is real-world big data of which it might not be practical to analyse using other methods. In a clinical development context data sources could include medical records, epidemiological or public health data, prior clinical study data, or IOT and IOB sensor data. Data scientists may combine data from multiple sources and types. Using analysis pipelines, machine learning techniques, neural networks, and decision tree analysis this data can be made sense of. The better the quality of the input data the more precise and accurate any predictive algorithms can be.
Statistical programmers
Statistical programmers help statisticians to efficiently clean and prepare data sets and mock TFLs in preparation for analysis. They set up SDTM and ADaM data structures in preparation for clinical studies. Quality control of data and advanced macros for database management are also key skills.
Biometricians
Biometricians use statistical methods to analyse data related to the characteristics of living organisms. They may work on topics such as growth patterns, reproductive success, or the genetic basis of traits. Biometricians may also be involved in developing new statistical methods for analysing data in these areas. Some use the terms biostatistician and biometrician interchangeably however for the purpose of this article they remain distinct.
Epidemiologists
Epidemiologists study the distribution and determinants of diseases in populations. Using descriptive, analytical, or experimental techniques, such as cohort or case-control studies, they identify risk factors for diseases, evaluate the effectiveness of public health interventions, as well as track or model the spread of infectious diseases. Epidemiologists use data from laboratory testing, field studies, and publicly available health data. They can be involved in developing new public health policies and interventions to prevent or control the spread of diseases.
Clinical trials and the role of data experts
Clinical trials involve testing new treatments, interventions, or diagnostic tests in humans. These studies are an important step in the process of developing new medical therapies and understanding the effectiveness and safety of existing treatments.
Biostatisticians are crucial to the proper design and analysis of clinical trials. So that optimal study design can take place, they may first have to conduct extensive meta-analysis of previous clinical studies and RWE generation based on available real-world data sets or R&D results. They may also be responsible for managing the data and ensuring its quality, as well as interpreting and communicating the results of the trial. From developing the statistical analysis plan and contributing to the study protocol, to final analysis and reporting, biostatisticians have a role to play across the project time-line.
During a clinical trial, statistical programmers may prepare data sets to CDISC standards and pre-specified study requirements, maintain the database, as well as develop and implement standard SAS code and algorithms used to describe and analyse the study data.
Bioinformaticians may be involved in the design and analysis stages of clinical trials, particularly if the trial design involves the use of large data sets such as sequencing data for multi-omics analysis. They may be responsible for managing and analysing this data, as well as developing software tools and algorithms to support the analysis.
Data scientists may be involved in designing and analysing clinical trials at the planning stage, as well as in developing new tools and methods. The knowledge gleaned from data science models can be used to improve decision-making across various contexts, including life sciences R&D and clinical trials. Some applications include optimising the patient populations used in clinical trials; feasibility analysis using simulation of site performance, region, recruitment and other variables, to evaluate the impacts of different scenarios on project cost and timeline.
Biometricians and epidemiologists may also contribute to clinical trials, particularly if the trial is focused on a specific population or on understanding the factors that influence the incidence or severity of a disease. They may contribute to the design of the study, collecting and analysing the data, or interpreting the results.
Overall, the role of these experts in clinical trials is to use their varied expertise in statistical analysis, data management, and research design to help understand the safety and effectiveness of new treatments and interventions.
The role of biostatistician in clinical trials
Biostatisticians may be responsible for developing the study protocol, determining the sample size, producing the randomisation schedule, and selecting the appropriate statistical methods for analysing the data. They may also be responsible for managing the data and ensuring its quality, as well as interpreting and communicating the results of the trial.
SDTM data preparation
The Study Data Tabulation Model (SDTM) is a data standard that is used to structure and organize clinical study data in a standardized way. Depending on how a CRO is structured, either biostatisticians, statistical programmers, or both will be involved in mapping the data collected in a clinical trial to the SDTM data set, which involves defining the structure and format of the data and ensuring that it is consistent with the standard. This helps to ensure that the data is organised in a way that is universally interpretable. This process involves working with the research team to ensure the appropriate variables and categories are defined before reviewing and verifying the data to ensure that it is accurate, complete and in line with industry standards. Typically the SDTM data set will be established early at the protocol phase and populated later once trial data is accumulated.
Creating and analysing the ADaM dataset
In clinical trials, the Analysis Data Model (ADaM) is a data set model used to structure and organize clinical trial data in a standardized way for the purpose of statistical analysis. ADaM data sets are used to store the data that will be analysed as part of the clinical trial, and are typically created from the Study Data Tabulation Model (SDTM) data sets, which contain the raw data collected during the trial. This helps to ensure the reliability and integrity of the data, and makes it easier to analyse and interpret the results of the trial.
Biostatisticians and statistical programmers are responsible for developing ADaM data sets from the SDTM data, which involves selecting the relevant variables and organizing them in a way that is appropriate for the particular statistical analyses that will be conducted. While statistical programmers may create derived variables, produce summary statistics, TFLs, and organise the data into appropriate datasets and domains, biostatisticians are responsible for conducting detailed statistical analyses of the data and interpreting the results. This may include tasks such as testing hypotheses, identifying patterns and trends in the data, and developing statistical models to understand the relationships between the data and the research questions the trial seeks to answer.
The role of biostatisticians, specifically, in developing ADaM data sets from SDTM data is to use their expertise in statistical analysis and research design to guide statistical programmers in ensuring that the data is organised, structured, and formatted in a way that is appropriate for the analyses that will be conducted, and to help understand and interpret the results of the trial.
A Biostatistician’s role in study design & planning
Biostatisticians play a critical role in the design, analysis, and interpretation of clinical trials. The role of the biostatistician in a clinical trial is to use their expertise in statistical analysis and research design to help ensure that the trial is conducted in a scientifically rigorous and unbiased way, and to help understand and interpret the results of the trial. Here is a general overview of the tasks that a biostatistician might be involved in during the different stages of a clinical trial:
Clinical trial design: Biostatisticians may be involved in designing the clinical trial, including determining the study objectives, selecting the appropriate study population, and developing the study protocol. They are responsible for determining the sample size and selecting the appropriate statistical methods for analysing the data. Often in order to carry out these tasks, preparatory analysis will be necessary in the form of detailed meta-analysis or systematic review.
Sample size calculation: Biostatisticians are responsible for determining the required sample size for the clinical trial. This is an important step, as the sample size needs to be large enough to detect a statistically significant difference between the treatment and control groups, but not so large that the trial becomes unnecessarily expensive or time-consuming. Biostatisticians use statistical algorithms to determine the sample size based on the expected effect size, the desired level of precision, and the expected variability of the data. This information is informed by expert opinion and simulation of the data from previous comparable studies.
Randomisation schedules: Biostatisticians develop the randomisation schedule for the clinical trial, which is a plan for assigning subjects to the treatment and control groups in a random and unbiased way. This helps to ensure that the treatment and control groups are similar in terms of their characteristics, which helps to reduce bias or control for confounding factors that might affect the results of the trial.
Protocol development: Biostatisticians are involved in developing the statistical and methodological sections of the clinical trial protocol, which is a detailed plan that outlines the objectives, methods, and procedures of the study. In addition to outlining key research questions and operational procedures the protocol should include information on the study population, the interventions being tested, the outcome measures, and the data collection and analysis methods.
Data analysis: Biostatisticians are responsible for analysing the data from the clinical trial, including conducting interim analyses and making any necessary adjustments to the protocol. They play a crucial role in interpreting the results of the analysis and communicating the findings to the research team and other stakeholders.
Final analysis and reporting: Biostatisticians are responsible for conducting the final analysis of the data and preparing the final report of the clinical trial. This includes summarising the results, discussing the implications of the findings, and making recommendations for future research.
The role of bioinformatician in biomarker-guided clinical studies.
Biomarkers are biological characteristics that can be measured and used to predict the likelihood of a particular outcome, such as the response to a particular treatment. Biomarker-guided clinical trials use biomarkers as a key aspect of the study design and analysis. In biomarker-guided clinical trials where the biomarker is based on genomic sequence data, bioinformaticians may play a particularly important role in managing and analysing the data. Genomic and other omics data is often large and complex, and requires specialised software tools and algorithms to analyse and interpret. Bioinformaticians develop and implement these tools and algorithms, as well as for managing and analysing the data to identify patterns and relationships relevant to the trial. Bioinformaticians use their expertise in computational biology to to help understand the relationship between multi-omics data and the outcome of the trial, and to identify potential biomarkers that can be used to guide treatment decisions.
Processing sequencing data is a key skill of bioinformaticians that involves several steps, which may vary depending on the specific goals of the analysis and the type of data being processed. Here is a general overview of the steps that a bioinformatician might take to process sequencing data:
Data pre-processing: Cleaning and formatting the data so that it is ready for analysis. This may include filtering out low-quality data, correcting errors, and standardizing the format of the data.
Mapping: Aligning the sequenced reads to a reference genome or transcriptome in order to determine their genomic location. This can be done using specialized software tools such as Bowtie or BWA.
Quality control: Checking the quality of the data and the alignment, and identifying and correcting any problems that may have occurred during the sequencing or mapping process. This may involve identifying and removing duplicate reads, or identifying and correcting errors in the data.
Data analysis: Using statistical and computational techniques to identify patterns and relationships in the data such as identifying genetic variants, analysing gene expression levels, or identifying pathways or networks that are relevant to the study.
Data visualization: Creating graphs, plots, and other visualizations to help understand and communicate the results of the analysis.
Once omics data has been analysed, the insights obtained can be used for tailoring therapeutic products to patient populations in a personalised medicine approach.
A changing role of data experts in life sciences R&D and clinical research
Due to the need for better therapies and health solutions, researchers are currently defining diseases at more granular levels using multi-omics insights from DNA sequencing data which allows differentiation between patients in the biomolecular presentation of their disease, demographic factors, and their response to treatment. As more and more of the resulting therapies reach the market the health care industry will need to catch up in order to provide these new treatment options to patients.
Even after a product receives regulatory approval, payers can opt not to reimburse patients, so financial benefit should be demonstrated in advance where possible. Patient reported outcomes and other health outcomes are becoming important sources of data to consider in evidence generation. Evidence provided to payers should aim to demonstrate financial as well as clinical benefit of the product.
In this context, regulators are becoming aware of the need for innovation in developing new ways of collecting treatment efficacy and other data used to assess novel products for regulatory approval. The value of observational studies and real-world-data sources as a supplement clinical trial data is being acknowledged as a legitimate and sometimes necessary part of the product approval process. Large scale digitisation now makes it easier to collect patient-centric data directly from clinical trial participants and users via devices and apps. Establishing clear evidence expectations from regulatory agencies then Collaborating with external stakeholders, data product experts, and service-providers to help build new evidence-building approaches.
Expert data governance and quality control is crucial to the success of any new methods to be implemented analytically. Data from different sources, such as IOT sensor data, electronic health records, sequencing data for multi-omics analysis, and other large data sets, has to be combined cautiously and with robust expert standards in place.
From biostatistics, bioinformatics, data science, CAS, and epidemiology for public heath or post-market modelling; a bespoke team of integrated data and analytics specialists is now as important to a product development project as the product itself to gaining competitiveness and therefore success in the marketplace. Such a team should be applying a combination of established data collection methodologies eg. clinical trials and systematic review, and innovative methods such as machine learning models that draw upon a variety of real world data sources to find a balance between advancing important innovation and mitigating risk.
The following article investigates several systematic reviews into sex and gender representation in individual clinical trial patient populations. In these studies sex ratios are assessed and evaluated by various factors such as clinical trial phase, disease type under investigation and disease burden in the population. Sex differences in the reporting of safety and efficacy outcomes are also investigated. In many cases safety and efficacy outcomes are pooled, rather than reported individually for each sex, which can be problematic when findings are generalised to the wider population. In order to get the dosage right for different body compositions and avoid unforeseen outcomes in off label use or when a novel therapeutic first reaches the market, it is important to report sex differences in clinical trials. Due to the unique nuances of disease types and clinical trial phases it is important to realise that a 50-50 ratio of male to female is not always the ideal or even appropriate in every clinical study design. Having the right sex balance in your clinical trial population will improve the efficiency and cost-effectiveness of your study. Based upon the collective findings a set of principles are put forth to guide the researcher in determining the appropriate sex ratio for their clinical trial design.
Sex difference by clinical trial phase
variation in sex enrolment ratios for clinical trial phases
females less likely to participate in early phases, due to increased risk of adverse events
under-representation of women in phase III when looking at disease prevalence
It has been argued that female representation in clinical trials is lacking, despite recent efforts to mitigate the gap. US data from 2000-2020 suggests that trial phase has the greatest variation in enrolment when compared to other factors, with median female enrolment being 42.9%, 44.8%, 51.7%, and 51.1% for phases I, I/II to II, II/III to III, and IV4. This shows that median female enrolment gradually increases as trials progress, with the difference in female enrolment between the final phases II/III to III and IV being <1%. Additional US data on FDA approved drugs including trials from as early as 1993 report that female participation in clinical trials is 22%, 48%, and 49% for trial phases I, II, and III respectively2. While the numbers for participating sexes are almost equal in phases II and III, women make up only approximately one fifth of phase I trial populations in this dataset2. The difference in reported participation for phase I trials between the datasets could be due to an increase in female participation in more recent years. The aim of a phase I trial is to evaluate safety and dosage, so it comes as no surprise that women, especially those of childbearing age, are often excluded due to potential risks posed to foetal development.
In theory, women can be included to a greater extent as trial phases progress and the potential risk of severe adverse events decreases. By the time a trial reaches phase III, it should ideally reflect the real-world disease population as much as possible. European data for phase III trials from 2011-2015 report 41% of participants being female1, which is slightly lower than female enrolment in US based trials. 26% of FDA approved drugs have a >20% difference between the proportion of women in phase II & III clinical trials and the prevalence of women in the US with the disease2, and only one of these drugs shows an over-representation of women.
Reporting of safety and efficacy by sex difference
Both safety and efficacy results tend to differ by sex.
Reporting these differences is inconsistent and often absent
Higher rates of adverse events in women are possibly caused by less involvement or non stratification in dose finding and safety studies.
There is a need to enforce analysis and reporting of sex differences in safety and efficacy data
Sex differences in response to treatment regarding both efficacy and safety have been widely reported. Gender subgroup analyses regarding efficacy can reveal whether a drug is more or less effective in one sex than the other. Gender subgroup analyses for efficacy are available for 71% of FDA approved drugs, and of these 11% were found to be more efficacious in men and 7% in women2. Alternatively, only 2 of 22 European Medicines Agency approved drugs examined were found to have efficacy differences between the sexes1. Nonetheless, it is important to study the efficacy of a new drug on all potential population subgroups that may end up taking that drug.
The safety of a treatment also differs between the sexes, with women having a slightly higher percentage (p<0.001) of reported adverse events (AE) than men for both treatment and placebo groups in clinical trials1. Gender subgroup analyses regarding safety can offer insights into the potential risks that women are subjected to during treatment. Despite this, gender specific safety analyses are available for only 45% of FDA approved drugs, with 53% of these reporting more side effects in women2. On average, women are at a 34% increased risk of severe toxicity for each cancer treatment domain, with the greatest increased risk being for immunotherapy (66%). Moreover, the risk of AE is greater in women across all AE types, including patient-reported symptomatic (female 33.3%, male 27.9%), haematologic (female 45.2%, male 39.1%) and objective non-haematologic (female 30.9%, male 29.0%)3. These findings highlight the importance of gender specific safety analyses and the fact that more gender subgroup safety reporting is needed. More reporting will increase our understanding of sex-related AE and could potentially allow for sex-specific interventions in the future.
Sex differences by disease type and burden
Several disease categories have recently been associated with lower female enrolment
Men are under-represented as often as women when comparing enrolment to disease burden proportions
There is a need for trial participants to be recruited on a case-by-case basis, depending on the disease.
Sex differences by disease type
When broken down by disease type, the sex ratio of clinical trial participation shows a more nuanced picture. Several disease categories have recently been associated with lower female enrolment, compared to other factors including trial phase, funding, blinding, etc4. Women comprised the smallest proportions of participants in US-based trials between 2000-2020 for cardiology (41.4%), sex-non-specific nephrology and genitourinary (41.7%), and haematology (41.7%) clinical trials4. Despite women being
proportionately represented in European phase III clinical studies between 2011-2015 for depression, epilepsy, thrombosis, and diabetes, they were significantly under-represented for hepatitis C, HIV, schizophrenia, hypercholesterolaemia, and heart failure and were not found to be overrepresented in trials for any of the disease categories examined1. This shows that the gap in gender representation exists even in later clinical trial phases when surveying disease prevalence, albeit to a lesser extent. Examining disease burden shows that the gap is even bigger than anticipated and includes the under-representation of both sexes.
Sex Differences by Disease Burden
It is not until the burden of disease is considered that men are shown to be under-represented as often as women. Including burden of disease can depict proportionality relative to the variety of disease manifestations between men and women. It can be measured as disability-adjusted life years (DALYs), which represent the number of healthy years of life lost due to the disease. Despite the sexes each making up approximately half of clinical trial participants overall in US-based trials between 2000-2020, all disease categories showed an under-representation of either women or men relative to disease burden, except for infectious disease and dermatologic clinical trials4. Women were under-represented in 7 of 17 disease categories, with the greatest under-representation being in oncology trials, where the difference between the number of female trial participants and corresponding DALYs is 3.6%. Men were under-represented compared with their disease burden in 8 of 17 disease categories, with the greatest difference being 11.3% for musculoskeletal disease and trauma trials.4 Men were found to be under-represented to a similar extent to women, suggesting that the under-representation of either sex could be by coincidence. Alternatively, male under-representation could potentially be due to the assumption of female under-representation leading to overcorrection in the opposite direction. It should be noted that these findings would benefit from statistical validation, although they illustrate the need for clinical trial participants to be recruited on a case-by-case basis, depending on the disease.
Takeaways to improve your patient sample in clinical trial recruiting:
Know the disease burden/DALYs of your demographics for that disease.
Try to balance the ratio of disease burden to the appropriate demographics for your disease
Aim to recruit patients based on these proportions
Stratify clinical trial data by the relevant demographics in your analysis. For example: toxicity, efficacy, adverse events etc should always be analyses separately for male and female to come up wit the respective estimates.
Efficacy /toxicity etc should always be reported separately for male and female. reporting difference by ethnicity is also important as many diseases differentially affect certain ethnicity and the corresponding therapeutics can show differing degrees of efficacy and adverse events.
The end goal of these is that medication can be more personalised and any treatment given is more likely to help and less likely to harm the individual patient.
Conclusions
There is room for improvement in the proportional representation of both sexes in clinical trials and knowing a disease demographic is vital to planning a representative trial. Assuming the under-representation is on the side of female rather than male may lead to incorrect conclusions and actions to redress the balance. Taking demographic differences in disease burden into account when recruiting trial participants is needed. Trial populations that more accurately depict the real-world populations will allow a therapeutic to be tailored to the patient.
Efficacy and safety findings highlight the need for clinical study data to be stratified by sex, so that respective estimates can be determined. This enables more accurate, sex/age appropriate dosing that will maximise treatment efficacy and patient safety, as well as minimise the chance of adverse events. This also reduces the risks associated with later off label use of drugs and may avoid modern day tragedies resembling the thalidomide tragedy. Moreover, efficacy and adverse events should always be reported separately for men and women, as the evidence shows their distinct differences in response to therapeutics.
1. Dekker M, de Vries S, Versantvoort C, Drost-van Velze E, Bhatt M, van Meer P et al. Sex Proportionality in Pre-clinical and Clinical Trials: An Evaluation of 22 Marketing Authorization Application Dossiers Submitted to the European Medicines Agency. Frontiers in Medicine. 2021;8.
2. Labots G, Jones A, de Visser S, Rissmann R, Burggraaf J. Gender differences in clinical registration trials: is there a real problem?. British Journal of Clinical Pharmacology. 2018;84(4):700-707.
3. Unger J, Vaidya R, Albain K, LeBlanc M, Minasian L, Gotay C et al. Sex Differences in Risk of Severe Adverse Events in Patients Receiving Immunotherapy, Targeted Therapy, or Chemotherapy in Cancer Clinical Trials. Journal of Clinical Oncology. 2022;40(13):1474-1486.
4. Steinberg J, Turner B, Weeks B, Magnani C, Wong B, Rodriguez F et al. Analysis of Female Enrollment and Participant Sex by Burden of Disease in US Clinical Trials Between 2000 and 2020. JAMA Network Open. 2021;4(6):e2113749.
Data sources for cost analysis of drug development R&D and clinical trials
Cost estimates for pre-clinical and clinical development across the pharmaceutical industry differ based on several factors. One of these is the source of data used by each costing study to inform these estimates. Several studies use private data, which can include confidential surveys filled out by pharmaceutical firms/clinical trial units and random samples from private databases3,9,10,14,15,16. Other studies have based their cost estimates upon publicly available data, such as data from the FDA/national drug regulatory agencies, published peer-reviewed studies, and other online public databases1,2,12,13,17.
Some have questioned the validity of using private surveys from large multinational pharmaceutical companies to inform cost estimates, saying that survey data may be artificially inflated by pharmaceutical companies to justify high therapeutic prices 18,19,20. Another concern is that per trial spending by larger pharmaceutical companies and multinational firms would far exceed the spending of start-ups and smaller firms, meaning cost estimates made based on data from these larger companies would not be representative of smaller firms.
Failure rate of R&D and clinical trial pipelines
Many estimates include the cost of failures, which is especially the case for cost estimates “per approved drug”. As many compounds enter the clinical trial pipeline, the cost to develop one approved drug/compound includes cost of failures by considering the clinical trial success rate and cost of failed compounds. For example, if 100 compounds enter phase I trials, and 2 compounds are approved, the clinical cost per approved drug would include the amount spent on 50 compounds.
The rate of success used can massively impact cost estimates, where a low success rate or high failure rate will lead to much higher costs per approved drug. The overall probability of clinical success may vary by year and has been estimated at a range of values including 7.9%21, 11.83%10, and 13.8%22. There are concerns that some studies suggesting lower success rates have relied on small samples from industry curated databases and are thereby vulnerable to selection bias12,22.
Success rates per phase transition also affects overall costs. When more ultimately unsuccessful compounds enter late clinical trial stages, the higher the costs are per approved compound. In addition, success rates are also dependent on therapeutic area and patient stratification by biomarkers, among other factors. For example, one study estimated the lowest success rate at 1.6% for oncological trials without biomarker use compared with a peak success rate of 85.7% for cardiovascular trials utilising biomarkers22. While aggregate success rates can be used to estimate costs, using specific success rates will be more accurate to estimate the cost of a specific upcoming trial, which could help with budgeting and funding decisions.
Out-of-pocket costs vs capitalised costs & interest rates
Cost estimates also differ due to reporting of out-of-pocket and capitalised costs. An out-of-pocket cost refers to the amount of money spent or expensed on the R&D of a therapeutic. This can include all aspects of setting up therapeutic development, from initial funding in drug discovery/device design, to staff and site costs during clinical trials, and regulatory approval expenses.
The capitalised cost of a new therapeutic refers to the addition of out-of-pocket costs to a yearly interest rate applied to the financial investments funding the development of a new drug. This interest rate, referred to as the discount rate, is determined by (and is typically greater than) the cost of capital for the relevant industry.
Discount rates for the pharmaceutical industry vary between sources and can dramatically alter estimates for capitalised cost, where a higher discount rate will increase capitalised cost. Most studies place the private cost of capital for the pharmaceutical industry to be 8% or higher23,24, while the cost of capital for government is lower at around 3% to 7% for developed countries23,25. Other sources have suggested rates from as high as 13% to as low as zero13,23,26, where the zero cost of capital has been justified by the idea that pharmaceutical firms have no choice but to invest in R&D. However, the mathematical model used in many estimations for the cost of industry capital, the CAPM model, tends to give more conservative estimates23. This would mean the 10.5% discount rate widely used in capitalised cost estimates may in fact result in underestimation.
While there is not a consensus on what discount rate to use, capitalised costs do represent the risks undertaken by research firms and investors. A good approach may be to present both out-of-pocket and capitalised estimated costs, in addition to rates used, justification for the rate used, and the estimates using alternative rates in a sensitivity analysis26.
Costs variation over time
The increase in therapeutic development costs
Generally, there has been a significant increase in the estimated costs to develop a new therapeutic over time26. One study reported an exponential increase of capitalised costs from the 1970s to the mid-2010s, where the total capitalised costs rose annually 8.5% above general inflation from 1990 to 201310. Recent data has suggested that average development costs reached a peak in 2019 and had decreased the following two years9. This recent decrease in costs was associated with slightly reduced cycle times and an increased proportion of infectious disease research, likely in response to the rapid response needed for COVID-19.
Recent cost estimates
Costs can range with more than 100-fold differences for phase III/pivotal trials alone1. One of the more widely cited studies on drug costs used confidential survey data from ten multinational pharmaceutical firms and a random sample from a database of publicly available data10. In 2013, this study estimated the total pre-approval cost at $2.6 billion USD per approved new compound. This was a capitalised cost, and the addition of post-approval R&D costs increased this estimate to $2.87 billion (2013 USD). The out-of-pocket cost per approved new compound was reported at $1.395 billion, of which $965 million were clinical costs and the remaining $430 million were pre-clinical.
Another estimate reported the average cost to develop an asset at $1.296 billion in 20139. Furthermore, it reported that this cost had increased until 2019 at $2.431 billion and had since decreased to $2.376 billion in 2020 and $2.006 billion in 2021. While comparable to the previous out-of-pocket estimate for 2013, this study does not state whether their estimates are out-of-pocket or capitalised, making it difficult to meaningfully compare these estimates.
Figure 1: Recent cost estimates for drug development per approved new compound. “Clinical only” costs include only the costs of phase 0-III clinical trials, while “full” costs include pre-clinical costs. The colour of each bubble indicates the study, while bubble size indicated relative cost. A dashed border indicated the study used private data for their estimations, while a solid border indicates the study utilised publicly available data. Figure represents studies 9, 10, 12, 13 and 17 from the reference list in this report.
Publicly available data of 63 FDA-approved new biologics from 2009-2018 was used to estimate the capitalised (at 10.5%) R&D investment to bring a new drug to market at median of $985.3 million and a mean of $1.3359 billion (inflation adjusted to 2018 USD)12. These data were mostly accessible from smaller firms, smaller trials, first-in-class drugs, and further specific areas. The variation in estimated cost was, through sensitivity analysis, mostly explained by success/failure rates, preclinical expenditures, and cost of capital.
Publicly available data of 10 companies with no other drugs on the market in 2017 was used to estimate out-of-pocket costs for the development of a single cancer drug. This was reported at a median of $648 million and a mean of $719.8 million13. Capitalised costs were also reported using a 7% discount rate, with a median of $754.4 million and mean of $969.4 million. By focusing on data from companies without other drugs on the market, these estimates may better represent the development costs per new molecular entity (NME) for start-ups as the cost of failure of other drugs in the pipeline were included while any costs related to supporting existing on-market drugs were systematically impossible to include.
One study estimated the clinical costs per approved non-orphan drug at $291 million (out-of-pocket) and $412 million (capitalised 10.5%)17. The capitalised cost estimate increased to $489 million when only considering non-orphan NMEs. The difference between these estimates for clinical costs and the previously mentioned estimates for total development costs puts into perspective the amount
spent on pre-clinical trials and early drug development, with one studynoting their pre-clinical estimates comprised 32% of out-of-pocket and 42% of capitalised costs10.
Things to consider about cost estimates
The issue with these estimates is that there are so many differing factors affecting each study. This complicates cost-based pricing discussions, especially when R&D cost estimates can differ orders of magnitude apart. The methodologies used to calculate out-of-pocket costs differ between studies9,17, and the use of differing data sources (public data vs confidential surveys) seem to impact these estimates considerably.
There is also an issue with the transparency of data and methods from various sources in cost estimates. Some of this is a result of using confidential data, where some analyses are not available for public scrutiny8. This study in particular raised questions as estimates were stated without any information about the methodology or data used in the calculation of estimates. The use of confidential surveys of larger companies has also been criticised as the confidential data voluntarily submitted would have been submitted anonymously without independent verification12.
Due to the limited amount of comprehensive and published cost data17, many estimates have no option but to rely on using a limited data set and making some assumptions to arrive at a reasonable estimate. This includes a lack of transparent available data for randomised control trials, where one study reported that only 18% of FDA-approved drugs had publicly available cost data18. Therefore, there is a need for transparent and replicable data in this field to allow for more plausible cost estimates to be made, which in turn could be used to support budget planning and help trial sustainability18,26.
Despite the issues between studies, the findings within each study can be used to gather an idea of trends, cost drivers, and costs specific to company/drug types. For example, studies suggest an increasing overall cost of drug development from 1970 to peak in 201910, with a subsequent decrease in 2020 and 20219.
Data evaluating the costs associated with developing novel therapeutics within the pharmaceutical industry can be used to identify trends over time and can inform more accurate budgeting for future research projects. However, the cost to develop a drug therapeutic is difficult to accurately evaluate, resulting in varying estimates ranging from hundreds of millions to billions of US dollars between studies. The high cost of drug development is not purely because of clinical trial expenses. Drug discovery, pre-clinical trials, and commercialisation also need to be factored into estimates of drug development costs.
There are limitations in trying to accurately assess these costs. The sheer number of factors that affect estimated and real costs means that studies often take a more specific approach. For example, costs will differ between large multinational companies with multiple candidates in their pipeline and start-ups/SMEs developing their first pharmaceutical. Due to the amount and quality of available data, many studies work mostly with data from larger multinational pharmaceutical companies with multiple molecules in the pipeline. When taken out of context, the “$2.6 billion USD cost for getting a single drug to market” can seem daunting for SMEs. It is very important to clarify what scale these cost estimates represent, but cost data from large pharma companies are still relevant for SMEs as they can used to infer costs for different scales of therapeutic development.
This mini-report includes what drives clinical trial costs, methods to reduce these costs, and then explores what can be learned from varying cost estimates.
What drives clinical trial costs?
There is an ongoing effort to streamline the clinical trial process to be more cost and time efficient. Several studies report on cost drivers of clinical trials, which should be considered when designing and budgeting a trial. Some of these drivers are described below:
Study size
Trial costs rise exponentially with an increasing study size, which some studies have found to be the single largest driver in trial costs1,2,3. There are several reasons for varying sample sizes between trials. For example, study size increases with trial phase progression as phases require different study sizes based on the number of patients needed to establish the safety and/or effectiveness of a treatment. Failure to recruit sufficient patients can result in trial delays which also increases costs4.
Trial site visits
A large study size is also correlated with a larger overall number of patient visits during a trial, which is associated with a significant increase in total trial costs2,3. Trial clinic visits are necessary for patient screening, treatment and treatment assessment but include significant costs for staff, site hosting, equipment, treatment, and in some cases reimbursement for patient travel costs. The number of trial site visits per patient varies between trials where more visits may indicate longer and/or more intense treatment sessions. One estimate for the number of trial visits per person was a median of 11 in a phase III trial, with $2 million added to estimated trial costs for every +1 to the median2.
Number & location of clinical trial sites
A higher number of clinical trial study sites has been associated with significant increase in total trial cost3. This is a result of increased site costs, as well as associated staffing and equipment costs. These will vary with the size of each site, where larger trials with more patients often use more sites or larger sites.
Due to the lower cost and shorter timelines of overseas clinical research5,6, there has been a shift to the globalisation of trials, with only 43% of study sites in US FDA-approved pivotal trials being in North America7. In fact, 71% of these trials had sites in lower cost regions where median regional costs were 49%-97% of site costs in North America. Most patients in these trials were either in North America (39.7%), Western Europe (21%), or Central Europe (20.4%).
Median cost per regional site as a percentage of North American median cost for comparison.
However, trials can face increased difficulties in managing and coordinating multiple sites across different regions, with concerns of adherence to the ethical and scientific regulations of the trial centre’s region5,6. Some studies have reported that multiregional trials are associated with a significant increase in total trial costs, especially those with sites in emerging markets3. It is unclear if this reported increase is a result of lower site efficiency, multiregional management costs, or outsourcing being more common among larger trials.
Clinical Trial duration
Longer trial duration has been associated with a significant increase in total trial costs3,4, where many studies have estimated the clinical period between 6-8 years8,9,10,11,12,13. Longer trials are sometimes necessary, such as in evaluating the safety and efficacy of long-term drug use in the management of chronic and degenerative disease. Otherwise, delays to starting up a trial contribute to longer trials, where delays can consume budget and diminish the relevance of research4. Such delays may occur as a result of site complications or poor patient accrual.
Another aspect to consider is that the longer it takes to get a therapeutic to market (as impacted by longer trials), the longer the wait is before a return of investment is seen by both the research organisation and investors. The period from development to on-market, often referred to as cycle time, can drive costs per therapeutic as interest based on the industry’s cost of capital can be applied to investments.
Therapeutic area under investigation
The cost to develop a therapeutic is also dependent on the therapeutic area, where some areas such as oncology and cardiovascular treatments are more cost intensive compared with others1,2,5,6,12,14. This is in part due to variation in treatment intensity, from low intensity treatments such as skin creams to high intensity treatments such as multiple infusions of high-cost anti-cancer drugs2. An estimate for the highest mean cost for pivotal trials per therapeutic area was $157.2M in cardiovascular trials compared to $45.4M in oncology, and a lowest of $20.8M in endocrine, metabolic, and respiratory disease trials1. This was compared to an overall median of $19M. Clinical
trial costs per therapeutic area also varied by clinical trial phase. For example, trials in pain and anaesthesia have been found to have the lowest average cost of a phase I study while having the highest average cost of a phase III study6.
It is important to note that some therapeutic areas will have far lower per patient costs when compared to others and are not always indicative of total trial costs. For example, infectious disease trials generally have larger sample sizes which will lead to relatively low per patient costs, whereas trials for rare disease treatment are often limited to smaller sample sizes with relatively high per patient costs. Despite this, trials for rare disease are estimated to have significantly lower drug to market costs.
Drug type being evaluated
As mentioned in the therapeutic areas section above, treatments may vary in intensity from skin creams to multiple rounds of treatment with several anti-cancer drugs. This can drive total trial costs due to additional manufacturing and the need for specially trained staff to administer treatments.
In the case of vaccine development, phase III/pivotal trials for vaccine efficacy can be very difficult to run unless there are ongoing epidemics for the targeted infectious disease. Therefore, some cost estimates of vaccine development include from the pre-clinical stages to the end of phase IIa, with the average cost for one approved vaccine estimated at $319-469 million USD in 201815.
Study design & trial control type used
Phase III trial costs vary based on the type of control group used in the trial1. Uncontrolled trials were the least expensive with an estimated mean of $13.5 million per trial. Placebo controlled trials had an estimated mean of $28.8 million, and trials with active drug comparators had an estimated mean cost of $48.9 million. This dramatic increase in costs is in part due to manufacturing and staffing to administer a placebo or active drug. In addition, drug-controlled trials require more patients compared to placebo-controlled, which also requires more patients than uncontrolled trials2.
Reducing therapeutic development costs
Development costs can be reduced through several approaches. Many articles recommend improvements to operational efficiency and accrual, as well as deploying standardised trial management metrics4. This could include streamlining trial administration, hiring experienced trial staff, and ensuring ample patient recruitment to reduce delays in starting and carrying out a study.
Another way to reduce development costs can take place in the thorough planning of clinical trial design by a biostatistician, whether in-house or external. Statistics consulting throughout a trial can help to determine suitable early stopping conditions and the most appropriate sample size. Sample size calculation is particularly important as underestimation undermines experimental results, whereas overestimation leads to unnecessary costs. Statisticians can also be useful during the pre-clinical stage to audit R&D data to select the best available candidates, ensure accurate R&D data analysis, and avoid pursuing unsuccessful compounds.
Other ways to reduce development costs include the use of personalised medicine, clinical trial digitisation, and the integration of AI. Clinical trial digitisation would lead to the streamlining of clinical trial administration and would also allow for the integration of artificial intelligence into clinical trials. There have been many promising applications for AI in clinical trials, including the use of electronic health records to enhance the enrolment and monitoring of patients, and the potential use of AI in trial diagnostics. More information about this topic can be found in our blog “Emerging use-cases for AI in clinical trials”.
Cost breakdown in more detail: How is a clinical trial budget spent?
Clinical trial costs can be broken down and divided into several categories, such as staff and non-staff costs. In a sample of phase III studies, personnel costs were found to be the single largest component of trial costs, consisting of 37% of the total, whereas outsourcing costs made up 22%, grants and contracting costs at 21%, and other expenses at 21%3.
From a CRO’s perspective, there are many factors that are considered in the cost of a pivotal trial quotation, including regulatory affairs, site costs, management costs, the cost of statistics and medical writing, and pass-through costs27. Another analysis of clinical trial cost factors determined clinical procedure costs made up 15-22% of the total budget, with administrative staff costs at 11-29%, site monitoring costs at 9-14%, site retention costs at 9-16%, and central laboratory costs at 4-12%5,6. In a study of multinational trials, 66% of total estimated trial costs were spent on regional tasks, of which 53.3% was used in trial sites and the remainder on other management7.
Therapeutic areas and shifting trends
Therapeutic area had previously been mentioned as a cost driver of trials due to differences in sample sizes and/or treatment intensity. It is however worth mentioning that, in 2013, the largest number of US industry-sponsored clinical trials were in oncology (2,560/6,199 active clinical trials with 215,176/1,148,340 patients enrolled)4,14. More recently, there has been a shift to infectious disease trials, in part due to the needed COVID-19 trials9.
Clinical trial phases
Due to the expanding sample size as a trial progresses, average costs per phase increase from phase I through III. Median costs per phase were estimated in 2016 at $3.4 million for phase I, $8.6 million for phase II, and $21.4 million for phase III3. Estimations of costs per patient were similarly most expensive in phase III at $42,000, followed by phase II at $40,000 and phase I at $38,50014. The combination of an increasing sample size and increasing per patient costs per phase leads to the drastic increase in phase costs with trial progression.
In addition, studies may have multiple phase III trials, meaning the median estimated cost of phase III trials per approved drug is higher than per trial costs ($48 million and $19 million respectively)2. Multiple phase III trials can be used to better support marketing approval or can be used for therapeutics which seek approval for combination/adjuvant therapy.
There are fewer cost data analyses available on phase 0 and phase IV on clinical trials. Others report that average Phase IV costs are equivalent to Phase III but much more variable5,6.
Orphan drugs
Drugs developed for the treatment of rare diseases are often referred to as orphan drugs. Orphan drugs have been estimated to have lower clinical costs per approved drug, where capitalised costs per non-orphan and orphan drugs were $412 million and $291 million respectively17. This is in part due to the limit to sample size imposed upon orphan drug trials by the rarity of the target disease and the higher success rate for each compound. However, orphan drug trials are often longer when compared to non-orphan drug trials, with an average study duration of 1417 days and 774 days respectively.
NMEs
New molecular entities (NMEs) are drugs which do not contain any previously approved active molecules. Both clinical and total costs of NMEs are estimated to be higher when compared to next in class drugs13,17. NMEs are thought to be more expensive to develop due to the increased amount of pre-clinical research to determine the activity of a new molecule and the increased intensity of clinical research to prove safety/efficacy and reach approval.
Conclusion & take-aways
There is no one answer to the cost of drug or device development, as it varies considerably by several cost drivers including study size, therapeutic area, and trial duration. Estimates of total drug development costs per approved new compound have ranged from $754 million12 to $2.6 billion10 USD over the past 10 years. These estimates do not only differ based on the data used, but also due to methodological differences between studies. The limited availability of comprehensive cost data for approved drugs also means that many studies rely on limited data sets and must make assumptions to arrive at a reasonable estimate.
There are still multiple practical ways that can be used to reduce study costs, including expert trial design planning by statisticians, implementation of biomarker-guided trials to reduce the risk of failure, AI integration and digitisation of trials, improving operational efficiency, improving accrual, and introducing standardised trial management metrics.
References
Moore T, Zhang H, Anderson G, Alexander G. Estimated Costs of Pivotal Trials for Novel Therapeutic Agents Approved by the US Food and Drug Administration, 2015-2016. JAMA Internal Medicine. 2018;178(11):1451-1457.
.1 Moore T, Zhang H, Anderson G, Alexander G. Estimated Costs of Pivotal Trials for Novel Therapeutic Agents Approved by the US Food and Drug Administration, 2015-2016. JAMA Internal Medicine. 2018;178(11):1451-1457.
2. Moore T, Heyward J, Anderson G, Alexander G. Variation in the estimated costs of pivotal clinical benefit trials supporting the US approval of new therapeutic agents, 2015–2017: a cross-sectional study. BMJ Open. 2020;10(6):e038863.
3. Martin L, Hutchens M, Hawkins C, Radnov A. How much do clinical trials cost?. Nature Reviews Drug Discovery. 2017;16(6):381-382.
4. Bentley C, Cressman S, van der Hoek K, Arts K, Dancey J, Peacock S. Conducting clinical trials—costs, impacts, and the value of clinical trials networks: A scoping review. Clinical Trials. 2019;16(2):183-193.
6. Sertkaya A, Wong H, Jessup A, Beleche T. Key cost drivers of pharmaceutical clinical trials in the United States. Clinical Trials. 2016;13(2):117-126.
7. Qiao Y, Alexander G, Moore T. Globalization of clinical trials: Variation in estimated regional costs of pivotal trials, 2015–2016. Clinical Trials. 2019;16(3):329-333.
10. DiMasi J, Grabowski H, Hansen R. Innovation in the pharmaceutical industry: New estimates of R&D costs. Journal of Health Economics. 2016;47:20-33.
11. Farid S, Baron M, Stamatis C, Nie W, Coffman J. Benchmarking biopharmaceutical process development and manufacturing cost contributions to R&D. mAbs. 2020;12(1):e1754999.
12. Wouters O, McKee M, Luyten J. Estimated Research and Development Investment Needed to Bring a New Medicine to Market, 2009-2018. JAMA. 2020;323(9):844-853.
13. Prasad V, Mailankody S. Research and Development Spending to Bring a Single Cancer Drug to Market and Revenues After Approval. JAMA Internal Medicine. 2017;177(11):1569-1575.
15. Gouglas D, Thanh Le T, Henderson K, Kaloudis A, Danielsen T, Hammersland N et al. Estimating the cost of vaccine development against epidemic infectious diseases: a cost minimisation study. The Lancet Global Health. 2018;6(12):e1386-e1396. 16. Hind D, Reeves B, Bathers S, Bray C, Corkhill A, Hayward C et al. Comparative costs and activity from a sample of UK clinical trials units. Trials. 2017;18(1).
17.Jayasundara K, Hollis A, Krahn M, Mamdani M, Hoch J, Grootendorst P. Estimating the clinical cost of drug development for orphan versus non-orphan drugs. Orphanet Journal of Rare Diseases. 2019;14(1).
19. Speich B, von Niederhäusern B, Schur N, Hemkens L, Fürst T, Bhatnagar N et al. Systematic review on costs and resource use of randomized clinical trials shows a lack of transparent and comprehensive data. Journal of Clinical Epidemiology. 2018;96:1-11.
20. Light D, Warburton R. Demythologizing the high costs of pharmaceutical research. BioSocieties. 2011;6(1):34-50.
21. Adams C, Brantner V. Estimating The Cost Of New Drug Development: Is It Really $802 Million?. Health Affairs. 2006;25(2):420-428.
23. Wong C, Siah K, Lo A. Estimation of clinical trial success rates and related parameters. Biostatistics. 2019;20(2):273-286. 24. Chit A, Chit A, Papadimitropoulos M, Krahn M, Parker J, Grootendorst P. The Opportunity Cost of Capital: Development of New Pharmaceuticals. INQUIRY: The Journal of Health Care Organization, Provision, and Financing. 2015;52:1-5. 25. Harrington, S.E. Cost of Capital for Pharmaceutical, Biotechnology, and Medical Device Firms. In Danzon, P.M. & Nicholson, S. (Eds.), The Oxford Handbook of the Economics of the Biopharmaceutical Industry, (pp. 75-99). New York: Oxford University Press. 2012. 26. Zhuang J, Liang Z, Lin T, De Guzman F. Theory and Practice in the Choice of Social Discount Rate for Cost-Benefit Analysis: A Survey [Internet]. Manila, Philippines: Asian Development Bank; 2007. Available from: https://www.adb.org/sites/default/files/publication/28360/wp094.pdf 27. Rennane S, Baker L, Mulcahy A. Estimating the Cost of Industry Investment in Drug Research and Development: A Review of Methods and Results. INQUIRY: The Journal of Health Care Organization, Provision, and Financing. 2021;58:1-11. 28. Ledesma P. How Much Does a Clinical Trial Cost? [Internet]. Sofpromed. 2020 [cited 26 June 2022]. Available from: https://www.sofpromed.com/how-much-does-a-clinical-trial-cost
Accurate sample size calculation plays an important role in clinical research. Sample size in this context simply refers to the number of human patients, wheather healthy or diseased, taking part in the study. Clinical studies conducted using an insufficient sample size can lack the statistical power to adequately evaluate the treatment of interest, whereas a superfluous sample size can unnecessarily waste limited resources.
Various methods can be applied for determining the optimal sample size for a specific clinical study. Methods also exist for any re-adjustments throughout the study, if required. These methods vary widely from straightforward tests and formulas to complex, time-consuming ones, depending on the type of study and available information from which to make the estimate. Most commonly used sample size calculation procedures are developed from a frequentist perspective
Importance of knowing your study parameters
Accurate sample size calculation requires, information on several key study and research parameters. These parameters usually include an effect size and variability estimate, derived from available sources; a clinically meaningful difference. In practice these parameters are generally unknown and must be estimated from the existing literature or from pilot studies.
The Bayesian Framework in sample size estimations and re-adjustments
The Bayesian Framework has gradually become one of the most frequently mentioned methods when it comes to randomised clinical trial sample size estimations and re-adjustments.
In practice, sample size calculation is usually treated explicitly as a decision problem and employs a loss or utility function.
The Bayesian approach involves three key stages:
1. Prior estimate
A researcher has a prior estimate about the treatment effect (and other study parameters) that has been derived from meta-analysis of existing research, pilot studies, or based on expert opinion in absence of these.
2. Likelihood
Data is simulated to derive a likelihood estimate of prior parameters.
3. Posterior estimate
Based on the insights obtained, prior estimates from the first stage are updated to give a more precise final estimate.
A challenge of using this approach is knowing when to stop this cycle when enough evidence has been gathered and avoid creating bias (Dreibe,2021). Peaking at the data in order to make a stopping decision is called “optional stopping”. In general an optional stopping rule is cautioned against as it can increase type one error rates (de Heide & Grunewald, 2021).
How to decide when to stop the simulation cycle?
There are two approaches one could take.
1. Posterior probability
Calculating the posterior probability that the mean difference between the treatment and control arm is equal or greater than the estimated effect of the intervention. Based on the level of probably calculated (low or high) the cycle could be stopped and without any further need to gather more data.
2. PPOS ( predictive probability of success)
Calculating the predictive probability of achieving a successful result at the end of the study is a commonly used approach. It is really helpful when it comes to determining the success or failure of a study. Similarly, as with posterior probability based on the level of probability a decision could be made to stop or continue the study.
How to plan a Bayesian sample size calculation for a clinical trial
The key elements to consider when planning a Bayesian clinical trial are the same as for frequentists clinical trial.
Key planning stages:
Determine the objective of the clinical study
Determine and set endpoints
Decide on the appropriate study design
Run a meta analysis or review of existing evidence related to your research objective
Statistical test and statistical analysis plan (SAP)
Even though the key planning stages are the same for both approaches it does not mean that they can be mixed through out the study. If you have chosen to use one approach you can’t change to another method once the calculations have been generated and research started.
Bayesian approach vs Frequentist approach for sample size calculations
Bayesian
Frequentist
Prior and posterior( uses probability of hypothesis and data)
No prior or posterior( never gives probability of hypothesis)
Sample size depends on the prior and likelihood
Sample size depend on the likelihood
Requeres finding/deciding on prior in order to estimate sample size
Does not require prior to estimate sample size
Computationally intensive due to integration over many parameters
Less computationally intense
Frequentist measures such as p-values and confidence intervals continue to predominate the methodology across life sciences research, however, the use of the Bayesian approach in sample size estimations and re-estimation for RTCs has been increasing over time.
Bayesian approach for sample size calculations in medical device clinical trial
In the recent years Bayesian approach has gained more popularity as the method used in clinical trials including medical device studies. One of the reasons being that if good prior information about the use of the specific therapeutic or device is available, the Bayesian approach may allow to include this information into the statistical analysis part of the clinical trial. Sometimes, the available prior information for a device of interest may be used as a justification for smaller sample size and shorten the length of the pivotal trial (Chen et al., 2011).
Computational algorithms and growing popularity of Bayesian approach
Bayesian statistical analysis can be computationally intense. Despite that there have been multiple breakthroughs with computational algorithms and increased computing speed that have made it much easier to calculate and build more realistic Bayesian models, further contributing to the popularity of Bayesian approach. (FDA, 2010).
Markov Chain Monte Carlo (MCMC) method
One of the basic computational tools being used is Markov Chain Monte Carlo ( MCMC) method. This method computes large number of simulations from the distributions of random quantities.
Why MCMC?
MCMC helps to deal with computational difficulties one often can face when using Bayesian approach for needed sample size estimations. The MCMC is an advanced random variable generation technique which allows one to simulate different samples from more sophisticated probability distributions.
Conclusion
Sample size calculation plays an important role in clinical research. If underestimated, statistical power for the detection of a clinically meaningful difference will likely be insufficient; if overestimated, resources are wasted unnecessarilly.
The Bayesian Framework has become quite popular approach for sample size estimation. There are advantages of using the Bayesian method, depite this there has been some criticism of this approach as a sample size estimation and re-adjustment method due to the prior being subjective and possibility of different researchers selecting different priors leading to different posteriors and final conclusions.
In reality, both the Bayesian and frequentist approaches to sample size calculation involve deriving the relevant input parameters from the literature or clinical expertise and could potentially differ due to variations in individual expert opinion as to which studies to include or exclude in this process.
Bayesian approach is more computationally intensive compared to the traditional frequentist approaches. Therefore, when it comes to selecting a method for sample size estimation, it should be chosen carefully to best fit the particular study design and base-on advice provided by statistical professionals with expertise in clinical trials.
References:
Bokai WANG, C., 2017. Comparisons of Superiority, Non-inferiority, and Equivalence Trials. [online] PubMed Central (PMC). Available at: <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5925592/> [Accessed 28 February 2022].
Chen, M., Ibrahim, J., Lam, P., Yu, A. and Zhang, Y., 2011. Bayesian Design of Noninferiority Trials for Medical Devices Using Historical Data. Biometrics, 67(3), pp.1163-1170.
E, L., 2008. Superiority, equivalence, and non-inferiority trials. [online] PubMed. Available at: <https://pubmed.ncbi.nlm.nih.gov/18537788/> [Accessed 28 February 2022].
Gubbiotti, S., 2008. Bayesian Methods for Sample Size Determination and their use in Clinical Trials. [online] Core.ac.uk. Available at: <https://core.ac.uk/download/pdf/74322247.pdf> [Accessed 28 February 2022].
U.S. Food and Drug Administration. 2010. Guidance for the Use of Bayesian Statistics in Medical Device Clinical. [online] Available at: <https://www.fda.gov/regulatory-information/search-fda-guidance-documents/guidance-use-bayesian-statistics-medical-device-clinical-trials> [Accessed 28 February 2022].
van Ravenzwaaij, D., Monden, R., Tendeiro, J. and Ioannidis, J., 2019. Bayes factors for superiority, non-inferiority, and equivalence designs. BMC Medical Research Methodology, 19(1).
de Heide. R, Grunewald, P.D, 2021, Why optional stopping can be a problem for Bayesians; Psychonomic Bulletin & Review, 21(2), 201-208.
