Knowledge Retrieval System for African Traditional Medicine: Enabling Drug Discovery from 1,000+ Research Papers

African traditional medicine encompasses centuries of plant-based healing knowledge scattered across thousands of unstructured research documents — largely inaccessible without manual literature review. Omdena partnered with a US-based drug discovery startup to build two proof-of-concept systems: a retrieval-augmented generation platform to make this literature queryable, and an AlphaFold API for protein structure prediction. The project delivered a working RAG system indexed on 1,000+ scientific documents and a functioning AlphaFold API, forming the AI integration stage of the client’s discovery platform.

Key Outcomes

The Problem: African Traditional Medicine Knowledge Buried in Unstructured Documents

African traditional medicine was one of the most extensive repositories of plant-based healing knowledge on the planet. Across the continent, remedies, compounds, and treatments passed down through oral traditions and documented in research literature represented a potential resource for affordable drug development, particularly for neglected and tropical diseases that received limited pharmaceutical investment.

The client’s challenge was foundational: this knowledge did not exist in a queryable form. Research on African traditional medicine was scattered across thousands of PDFs, and practitioners could not retrieve relevant evidence without conducting manual literature reviews across hundreds of documents. Building a drug discovery platform required solving the knowledge access problem first.

Beyond knowledge retrieval, the client needed a computational tool to model the molecular structures of traditional compounds, a prerequisite for simulating how plant-based molecules could be combined to discover potential new treatments. No existing tool connected African traditional medicine research to compound structure modeling at this scale, making protein structure prediction a second foundational challenge.

The Partnership: Omdena and a US-Based Drug Discovery Startup

Omdena partnered with the client, a technology company building an AI-powered discovery platform for African traditional medicine. The client’s broader mission encompassed digitizing traditional medical knowledge from community sources, including oral histories, handwritten notes, voice recordings, and field observations, and using AI to identify plant-based compounds and simulate how they could be combined to discover new treatments for neglected and tropical diseases.

Omdena contributed the AI development team to build two proof-of-concept systems: the RAG knowledge retrieval pipeline and the AlphaFold protein structure prediction API, together forming the AI integration layer of the platform. The PoC focused on the scientific literature corpus: over 1,000 research papers collected to seed the knowledge base before community-sourced data collection was integrated.

Why Building AI for African Traditional Medicine Discovery Is Hard

• Multi-format source material: African traditional medicine knowledge spanned oral histories, voice recordings, handwritten field notes, and scientific research papers — each format requiring different extraction and transcription methods. The PoC focused on the scientific literature layer, where 1,000+ PDFs presented varied layouts, encoding inconsistencies, and noisy extracted text that required dedicated preprocessing before retrieval could operate.
• Domain-specific terminology: Traditional medicine research combined botanical Latin names, vernacular plant identifiers, pharmacological compound descriptions, and clinical observation language within the same document. Standard NLP tokenization and stopword lists trained on general corpora required careful adaptation to handle this mixed-register text without discarding domain-critical terms.
• Hallucination risk in AI generation: A knowledge system used to inform drug discovery decisions could not generate answers that were not traceable to specific retrieved documents. Standard language model generation produced fluent but potentially unverifiable responses. Retrieval-augmented generation was the only architecture that grounded every output in actual retrieved passages — making it the correct choice for this domain.
• Protein structure prediction without existing templates: Predicting the three-dimensional structure of plant-based compounds required running AlphaFold on protein sequences from traditional remedy research. Standard protein databases contained limited coverage of compounds specific to African traditional medicine, requiring the system to predict novel structures without relying solely on template matching.

Building the Materia Medica Knowledge Platform: RAG System and AlphaFold Protein Structure Prediction

Step 1 — PDF Extraction

The pipeline ingested over 1,000 scientific PDFs using pdfplumber, which handled varied document layouts and extracted text while preserving structural information such as section headers and paragraph boundaries. Documents were processed from a zip archive, and raw text was passed to the NLP preprocessing stage. Inconsistent encoding and layout variation across papers from multiple publishers was the primary challenge.

Step 2 — NLP Preprocessing

Extracted text was preprocessed using NLTK. The pipeline applied tokenization to segment text into discrete units, removed stopwords carrying no retrieval signal, and applied lemmatization to normalize term variants to root forms. This step was critical to retrieval quality: noisy extraction output produced degraded vector embeddings, and preprocessing quality had a greater effect on retrieval performance than tuning the retrieval model.

Step 3 — FAISS Vector Indexing

Preprocessed text was encoded into dense vectors using SentenceTransformer and indexed with FAISS (Facebook AI Similarity Search). FAISS enabled millisecond-scale similarity search across the document corpus by comparing query embeddings against stored vectors. Dense Passage Retrieval (DPR) was used alongside FAISS to retrieve the most semantically relevant passages — enabling meaning-level retrieval beyond keyword matching.

Step 4 — RAG Model Integration

Retrieved documents were passed to the Hugging Face RAG model, which combined BART/T5 generation with retrieved context. The model was conditioned on retrieved passages rather than parametric knowledge alone, grounding every response in actual documents. This architecture reduced the risk of generating claims not traceable to the underlying literature, which mattered when outputs might inform drug discovery decisions.

Step 5 — Question-Answering and Output Formatting

The final stage managed the complete QA pipeline using Hugging Face Transformers and the OpenAI API to format and deliver answers to the user. The output layer structured generated responses for readability and ensured that the final answer remained traceable to the retrieved source documents that grounded it.

The AlphaFold System: Protein Structure Prediction for Traditional Compounds

Alongside the RAG system, the project built a parallel AlphaFold proof-of-concept for protein structure prediction. FastAPI endpoints accepted protein sequences, Pydantic-validated pipelines processed them, and ColabFold with Mamba and PDBFixer ran structure prediction with optional Amber relaxation. Results were saved as PDB files and served through a Uvicorn web interface — establishing the compound structure modeling layer of the client’s discovery platform.

What Was Delivered: RAG Knowledge Retrieval System and AlphaFold Protein Structure Prediction API

The project delivered two parallel proof-of-concept systems: a functioning end-to-end RAG pipeline indexed on over 1,000 scientific papers, and an AlphaFold protein structure prediction API built with FastAPI, ColabFold, and Mamba. Together they formed the AI integration layer of the client’s discovery platform, connecting knowledge retrieval from traditional medicine literature to computational compound structure modeling.

Both systems demonstrated that the RAG and AlphaFold architectures were viable for traditional medicine applications and established the technical foundation for the client’s broader discovery platform. Every component of both pipelines was built, documented, and demonstrated end-to-end: PDF extraction through RAG generation for the knowledge system, and sequence input through PDB file output for the AlphaFold system.

Impact: From Manual Literature Review to Grounded AI Answers and Computational Compound Modeling

Before the project, accessing the body of research on African traditional medicine required manual search across hundreds of individual PDF documents, a process that could take hours for a single research question. The proof of concept demonstrated that a conversational query interface could surface relevant evidence from over 1,000 papers in seconds, with each answer traceable to the source documents that grounded it.

The two systems together established the AI integration stage of the client’s seven-stage AI Discovery Platform, which aimed to digitize traditional medicine knowledge from community sources, including oral histories, voice recordings, and field observations, through to drug discovery and development for neglected and tropical diseases. Both PoCs demonstrated that this stage of the platform was technically feasible before committing to production implementation.

For communities whose traditional knowledge fed into the platform, the longer-term model anticipated that pipeline profits would return to communities via NGOs, supporting infrastructure, education, and economic development alongside healthcare innovation for the diseases that affected them most.

Benefits for Clients: Three Capabilities That Did Not Exist Before

• Queryable knowledge base over 1,000+ research documents: Researchers and practitioners gained access to a structured, searchable system over scientific literature that previously existed only as isolated, unconnected PDFs. A query that would have required hours of manual literature review could be answered in seconds with a grounded, formatted response drawn from the indexed corpus.
• Plain-language query interface requiring no NLP expertise: The question-answering layer allowed users to submit queries in natural language without any understanding of the underlying vector search or retrieval-augmented generation systems. Accessing the knowledge base required no specialist data science or machine learning skills — only the ability to phrase a question.
• Open-source, auditable pipelines via OmdenaLore: Both pipelines — the RAG system from PDF extraction through FAISS indexing to generation, and the AlphaFold system from sequence input to PDB output —were documented and made available through OmdenaLore, Omdena’s open-source knowledge-sharing platform. Both codebases could be audited, extended, and adapted to other traditional medicine knowledge domains without re-engineering from scratch.

What Building a Dual-System AI Platform for Traditional Medicine Knowledge Taught Us

Preprocessing quality determines retrieval quality more than the retrieval model does.

The FAISS and DPR retrieval components performed well only when input text was clean and consistently normalized. Less thorough NLTK preprocessing produced degraded embeddings and lower-quality retrieval results. Investing time in tokenization, stopword removal, and lemmatization had a greater effect on system quality than tuning the retrieval model — a result that applied broadly to RAG systems across domains.

Retrieval-augmented generation is the correct architecture when outputs carry real-world consequences.

A language model trained on general corpora could generate fluent responses about plant compounds and traditional remedies — but those responses might not correspond to anything in the actual literature. For a system that might inform drug discovery decisions, grounding every answer in retrieved documents was not optional. The RAG architecture was selected not for performance alone but for accountability and traceability.

Building two parallel PoCs exposes integration constraints that single-system development would miss.

The PoC demonstrated the viability of both the RAG and AlphaFold systems and established the core pipelines, but also identified the gaps a production system would need to address: domain-specific fine-tuning of the embedding model, handling oral and handwritten sources requiring ASR and OCR preprocessing, and integrating the two PoC systems into a unified production pipeline.

What’s Next: Expanding the Knowledge Base and Integrating Protein Structure Prediction

The two systems built in this project represented the AI integration stage of the client’s seven-stage discovery platform. Immediate next steps included expanding the document corpus beyond 1,000 papers, fine-tuning the SentenceTransformer embedding model on domain-specific botanical and pharmacological terminology, and extending the preprocessing pipeline to handle oral and handwritten source materials through ASR and OCR integration.

The longer roadmap focused on integrating the RAG and AlphaFold systems into a unified pipeline, enabling the platform to move from surfacing evidence about traditional compounds to predicting their molecular structures in a single workflow. Beyond the technical integration, the platform aimed to establish a community benefit model in which profits from drug development flowed back to the communities whose traditional knowledge seeded the system.

Key Achievements: Three Deliverables from the Materia Medica Proof of Concept

1,000+ Document Knowledge Base Built from Scientific Literature

The team ingested, extracted, preprocessed, and indexed over 1,000 scientific papers on African traditional medicine, building from scratch a structured knowledge corpus that had not previously existed in any queryable format. The knowledge base covered traditional remedies, plant compounds, and clinical observations spanning multiple conditions and disease areas relevant to the client’s discovery platform.

End-to-End RAG Pipeline from PDF to Formatted Answer

The five-stage pipeline — PDF extraction, NLP preprocessing, FAISS vector indexing, RAG generation, and QA formatting — was built and demonstrated as a working end-to-end system. The team documented each stage and designed the pipeline for extensibility: the corpus could grow, and the retrieval model could be fine-tuned without rebuilding the core architecture.

AlphaFold Protein Structure Prediction API Built as a Parallel Proof of Concept

The team built a second PoC system alongside the RAG pipeline: an AlphaFold-based protein structure prediction API using FastAPI, ColabFold, Mamba, and PDBFixer with Amber relaxation. The system accepted protein sequences, ran structure prediction, saved outputs as PDB files, and served results through a web interface, establishing the compound modeling layer the RAG system was designed to feed.

About the Project

Omdena partnered with a US-Based Drug Discovery Startup to build two proof-of-concept systems for African traditional medicine research: a RAG pipeline indexed on 1,000+ scientific papers spanning PDF extraction, NLTK preprocessing, FAISS vector search, and Hugging Face RAG generation; and an AlphaFold protein structure prediction API built with FastAPI, ColabFold, and Mamba. Both systems formed the AI integration stage of the client’s discovery platform.