Latent Dirichlet Allocation in predicting clinical trial terminations

Area of Science:

• Computational linguistics
• Biomedical informatics
• Machine learning in healthcare

Background:

• A significant percentage of funded research studies terminate without successful outcomes.
• The reasons behind study terminations are not well understood, representing a knowledge gap.
• High termination rates are concerning given the rigorous planning and peer-review of funded studies.

Purpose of the Study:

• To identify factors contributing to clinical study failures.
• To distinguish between studies that complete successfully and those that terminate.
• To leverage natural language processing and machine learning for predicting study outcomes.

Main Methods:

• Utilized data from ClinicalTrials.gov, including structured and unstructured narrative data.
• Applied Latent Dirichlet Allocation (LDA) to derive 25 topics from narrative data.
• Developed random forest models using structured data alone and combined with LDA-derived topics to predict study termination.

Main Results:

• Latent Dirichlet Allocation (LDA) demonstrated significant interpretive and predictive value for clinical trial failure.
• A combined modeling approach using structured data and text topics yielded robust predictive probabilities.
• The model incorporating text topics showed improved sensitivity and specificity compared to using structured data alone.

Conclusions:

• Topic modeling with LDA enhances the utility of unstructured data for predicting study completion versus termination.
• This research provides a foundation for future studies evaluating the viability of health study designs.
• The findings suggest a novel approach to proactively identify and potentially mitigate risks of clinical trial failure.