AI-Powered Drug Repurposing: Using NLP to Extract Clinical Evidence from 584 Cancer Trial Abstracts

Cancer generates 17 million new diagnoses and 10 million deaths annually, yet clinical evidence supporting generic drug repurposing is buried in thousands of unstructured research abstracts. Omdena partnered with a US-based drug repurposing nonprofit, a tech nonprofit accelerating access to repurposed cancer treatments, and deployed 50 contributors to build NLP pipelines extracting clinical outcome measures from randomised controlled trial abstracts. The project delivered a structured outcome database across 584 abstracts, a curated 50-abstract annotation dataset, and three extraction approaches covering rule-based NER, Python Regex, and semi-supervised machine learning

Key Outcomes

The Problem: Clinical Evidence for Affordable Drug Repurposing Is Buried in Unstructured Text

Cancer remains one of the most resource-intensive conditions in global healthcare, with 17 million new diagnoses, 10 million deaths, and approximately one trillion dollars in annual costs. A significant share of those costs comes from the price of cancer-specific treatments, even where cheaper generic drugs have demonstrated clinical effectiveness in published research.

Repurposing approved non-cancer generic drugs for oncology use offers a direct route to more affordable treatment. Acting on that opportunity requires locating the clinical evidence, which exists across thousands of randomized controlled trial abstracts but is recorded in unstructured narrative text rather than any standardized, queryable format.

The client faced this bottleneck directly. Extracting relevant outcome measures — patient counts, response rates, survival durations, drug names — from each abstract manually was not feasible at the volume required to survey the clinical literature systematically. The project objective was to automate this extraction at scale.

The Partnership: Omdena and a US-Based Drug Repurposing Nonprofit

The client is a US-based drug repurposing nonprofit whose core mission is identifying and accelerating access to affordable, already-approved generic drugs that clinical evidence suggests could be repurposed for cancer treatment. The organization maintains a research program that draws on randomized controlled trial literature to evaluate which off-patent drugs show sufficient clinical evidence to merit further investigation.

Omdena contributed a team of 50 data science contributors to the project. The source material provided by the client consisted of clinical study abstracts containing study IDs, titles, and the abstract text — the primary input for all annotation, extraction, and modeling work conducted across the project.

Why Clinical NLP Is Hard: Terminology, Notational Variation, and Rare Entities

Clinical research abstracts present structural and linguistic challenges that general-purpose NLP approaches were not designed to handle. Three categories of difficulty shaped every design decision the team made and explained why standard NLP rules that work reliably on general text routinely fail on clinical literature:

• Notational inconsistency: the same clinical outcome appears under dozens of abbreviations and phrasings across different publications. Progression-free survival is reported as PFS, progression-free survival, time to progression, and numerous other variants. There is no standard format across the RCT literature for recording OS, PFS, response rates, or hazard ratios.
• Rare entity classes: drug names such as bevacizumab or temozolomide appear in only a subset of the dataset. Rules and models optimized for frequently occurring entities do not transfer reliably to low-frequency mentions, and insufficient training examples make supervised learning approaches harder to apply.
• Entity interdependency: a response rate figure is only meaningful when linked to the correct patient group, treatment arm, and drug. Extracting values in isolation produces records that cannot be reliably interpreted. The pipeline needed to capture entities and their relational context, not just isolated values.

Building the Pipeline: Annotation, Rule-Based NER, Regex, and Semi-Supervised Learning

Step 1: Annotation Layer and Ground Truth Dataset

The first step was building a validated annotation dataset to serve as ground truth for evaluating extraction approaches. The team configured Labelbox with Model-Assisted Labeling enabled, allowing model-generated pre-annotations to be loaded into the platform before human reviewers assessed each abstract. A pre-annotation API, deployed on Heroku, managed the transfer of model outputs back into Labelbox.

Approximately 90 abstracts were annotated in the initial pass. After a two-stage review involving Omdena contributors first and then the client team, 50 abstracts were confirmed as high quality and usable for model evaluation. A consensus metric tracked labeler agreement on each abstract, measuring index overlap across annotators to surface disagreements before final validation.

Step 2: spaCy Rule-Based Named Entity Recognition

The first extraction approach used spaCy’s EntityRuler pipeline to define custom pattern rules for named entity recognition. Rules targeted drug names using exact-match patterns for known compounds including hydroxychloroquine, bevacizumab, pembrolizumab, temozolomide, and others, alongside token-attribute patterns to capture sample size expressions and clinical outcome values from the abstract text.

Step 3: Python Regex NER

A parallel extraction approach using Python’s Regex library captured eight specific clinical outcome measures across all 584 abstracts in the RCT dataset. The regex model extracted 102 Group 1 and 72 Group 2 response rate values, 58 and 55 median overall survival figures, 45 and 43 median progression-free survival values, and 81 and 61 survival rate figures, respectively.

Hazard ratio and p-value extraction was attempted under both the spaCy and regex approaches. Both entity types showed too much notational variation across abstracts for rule-based methods to capture reliably. HR and p-value results were excluded from the final output and documented as targets for a future ML-based extraction phase where learned representations can handle the variation that fixed rules cannot.

Step 4: Machine Learning and Semi-Supervised Approaches

Machine learning approaches were explored to complement the two rule-based methods. The primary approach was Unsupervised Data Augmentation, designed to improve model performance where only a small labeled dataset was available, using the 50-abstract annotation set more effectively than standard supervised training alone.

A rule-based model using Python Regex provided an interpretable baseline against which the ML approaches were compared. The parallel development of rule-based and ML approaches was intentional: each method had different strengths on different entity types, and the architecture was designed to allow the best-performing approach for each entity to be selected independently.

What Was Delivered: Structured Outcome Database Across 584 RCT Abstracts

Table 1: Schema of the structured outcome database delivered across 584 RCT abstracts. Each row in the delivered database represents one processed abstract. — indicates the entity was not reported in that abstract. Hazard ratios and p-values are excluded across all records due to notational variation too high for rule-based extraction.

The pipeline was applied to all 584 randomized controlled trial abstracts in the project dataset, extracting and structuring predefined clinical outcome measures at a scale that manual review could not have matched within a comparable timeline. Three outputs were delivered:

• Structured outcome database containing extracted values for drug names, sample sizes, median OS, median PFS, response rates, and survival rates across 584 RCT abstracts — with unreliable entities excluded and documented
• Curated 50-abstract annotation dataset with validated ground truth labels, reviewed by Omdena contributors and the client team, suitable for training and evaluating future extraction models
• Pre-annotation API enabling new abstracts to be pre-labeled by the model before human review in Labelbox, reducing future annotation overhead without removing the human review step.

For entities where rule-based extraction proved reliable, the database gave the client structured, queryable records of clinical outcomes that previously existed only as narrative text. Hazard ratios and p-values were excluded from the delivered output and documented as the priority target for the next extraction phase.

Impact: A Queryable Evidence the Base Client Did Not Have Before

The primary impact is structural. Before this project, the client’s research program depended on manual review of individual clinical abstracts to identify which generic drugs showed meaningful outcomes in cancer trials. The NLP pipeline converted that abstract-by-abstract manual process into an automated extraction workflow that operates across the full 584-abstract dataset simultaneously.

A structured, queryable database of outcome measures changed what was analytically possible. The client could filter across drug names, response rate thresholds, survival durations, and patient group sizes across 584 abstracts at once, rather than reading each abstract individually to locate comparable data points. This shifted the research workflow from extraction to analysis.

The pipeline was also reusable and extendable. New RCT abstracts could be processed through the same annotation and extraction workflow, incrementally extending the database as the clinical literature grew without rebuilding the system. The pre-annotation API reduced the marginal cost of adding each new batch of abstracts.

Benefits for the Client: Speed, Scalability, and a Reusable Annotation Asset

• Automated extraction at literature scale:the NLP pipeline processed abstracts in a fraction of the time required for manual review, enabling the client to survey the RCT literature at a volume and speed that was not operationally feasible before.
  
• A reusable ground truth dataset: the 50-abstract annotation dataset constituted a validated resource for training and evaluating future extraction models. As the annotation volume grows, so does the ceiling on what supervised and semi-supervised models can achieve.
• Extendable without rebuilding:the three-approach pipeline covering rule-based NER, regex, and ML was designed so individual components could be replaced or upgraded independently, accommodating new entity types and drug classes as the client’s research program expands its scope.
  

What Building This NLP Pipeline for Cancer Drug Discovery Taught Us

A rule-based extraction system cannot be properly evaluated without a ground truth dataset built alongside it from the start.

The regex and spaCy models extracted entities from 584 abstracts, but measuring their precision and recall required a complete ground truth set that was not available for the full dataset at project completion. Building the annotation dataset in parallel with the extraction models, rather than sequentially, would have enabled formal performance evaluation from the first deployment rather than at the end.

Annotation quality determines the model ceiling: agreement metrics need to measure semantic content, not just index overlap.

The consensus metric used to track labeler agreement measured index overlap between annotations rather than whether annotators agreed on the semantic content of the labeled entity. High consensus scores therefore did not guarantee label precision, and a second manual review layer was required to catch disagreements that the automated metric missed. Semantic agreement metrics would have surfaced these issues earlier.

Not all clinical entities are extractable with rule-based methods: some require learned representations that can handle notational variation.

Hazard ratio and p-value extraction was attempted across both the regex and spaCy rule sets and excluded from the final output because notational variation across abstracts was too high for fixed rules to handle reliably. These entities require an ML approach where the model learns from examples across varied notation rather than matching against a fixed pattern set.

What’s Next: Ground Truth Completion, Full-Paper Extraction, and Expanded Entity Coverage

The immediate next step is completing a full ground truth annotation dataset, which will enable formal precision and recall evaluation of all three extraction approaches across the 584-abstract corpus. With a validated evaluation set, the best-performing approach for each entity type can be selected and promoted to the primary extraction model for that entity class.

Longer-term development includes extending the pipeline from abstract text to full paper content, which contains richer outcome data but significantly more noise and formatting variation. Expanding the entity set to cover hazard ratios, p-values, and additional drug classes not included in the initial scope would broaden database coverage across the repurposing literature.

Key Achievements: Structured Database, Annotation Dataset, and Three-Approach NLP Pipeline

Structured outcome database extracted from 584 randomized controlled trial abstracts

The NLP pipeline processed all 584 RCT abstracts in the project dataset and produced a structured database of predefined outcome measures. The database captures drug names, sample sizes, median overall survival, median progression-free survival, response rates, and survival rates, enabling queryable access to clinical data that previously existed only in unstructured narrative text.

Curated 50-abstract annotation dataset validated through two-stage expert review

Fifty clinical study abstracts were annotated in Labelbox and validated by Omdena contributors and the client team through a two-stage review process. The dataset constitutes a reusable ground truth resource for training and evaluating future NLP models, with annotation quality enforced through both the automated consensus metric and manual correction where the metric was insufficient.

Three-approach NLP extraction system covering rule-based NER, Regex, and semi-supervised ML

The project delivered three distinct extraction methodologies: a spaCy EntityRuler pipeline for pattern-based NER, a Python Regex system that extended entity coverage to eight additional clinical outcome types, and an Unsupervised Data Augmentation approach for low-labeled-data conditions. Each approach is documented and independently extendable as the client’s entity set expands.

About This Project

This case study documents Omdena’s collaboration with a US-based drug repurposing nonprofit focused on repurposing affordable generic drugs for cancer treatment. A team of 50 Omdena contributors built NLP and machine learning pipelines to extract and structure predefined clinical outcome measures from 584 randomized controlled trial abstracts, delivering a structured outcome database, a validated annotation dataset, and a pre-annotation API.