Document Parsing for RAG: Handling Multi-Column Documents

Retrieval-Augmented Generation (RAG) is an AI approach that improves large language models by retrieving relevant information from external documents before generating an answer. Instead of relying solely on training data, RAG systems ground their responses in real source material, such as PDFs, reports, or manuals. However, the effectiveness of RAG depends heavily on one often overlooked step: document parsing.

Modern organizations rely on complex documents, including text-heavy PDFs, multi-column reports, tables, and scanned files. If these documents are parsed incorrectly, retrieval breaks down, and LLMs receive fragmented or misleading context. 

This article explains how you can parse complex documents reliably in RAG systems. It also explores common parsing failures, layout-aware techniques, modern tools, and practical strategies to improve retrieval accuracy and reduce hallucinations.

Why Document Parsing for RAG Is Critical

In Retrieval-Augmented Generation systems, most teams spend their time choosing the best language model or vector database. But many RAG systems fail long before those components ever matter. The real problem often starts with document parsing. 

If documents are parsed incorrectly, the retrieval step pulls messy, incomplete, or out-of-order text. When that happens, the language model can only produce poor answers or make things up. This is the classic “garbage in, garbage out” problem.

Parsing mistakes spread through the entire pipeline, hurting chunking, indexing, and retrieval relevance. That’s why parsing deserves serious attention in RAG design. To see why this is so challenging, it’s important to understand the types of complex documents RAG systems deal with in real-world scenarios.

Complex Document Types RAG Needs to Handle

Not all PDFs behave the same way. While simple, single-column documents can often be handled by basic text extractors, real-world RAG systems must work with far more complex inputs. Production pipelines frequently encounter text-rich, visually structured documents, including:

• Research papers and academic journals
• Legal contracts and policy documents
• Government and financial reports
• Technical manuals and specifications

These documents have shared traits that make parsing difficult:

• High information density
• Multi-column layouts
• Clear hierarchical structure with titles, sections, and lists
• Interleaved elements such as tables, figures, and captions
• Reading order defined by visual layout rather than linear text flow

In these documents, meaning is spatial, not sequential. Content is arranged across columns and visual blocks that determine reading order. Most RAG pipelines ignore this structure, so traditional text extraction breaks down as soon as layouts become complex.

Why Traditional PDF Extraction Fails?

Most RAG pipelines rely on basic PDF loaders such as PyPDF or PDFMiner. These tools flatten documents into plain text, ignoring layout, columns, and semantic block types like titles or tables. This works for simple PDFs but fails on text-rich documents. In multi-column layouts, standard parsers read across the page, mixing content from different columns. The result is jumbled text with lost meaning, which embedding models cannot fix.

The most common failures include:

• Broken reading order: text from different columns gets mixed together
• Lost hierarchy: headings and sections collapse into plain text
• Poor segmentation: chunks split mid-sentence or merge unrelated content
• Visual noise: headers, footers, and page numbers leak into the text

In RAG systems, these issues compound downstream and severely degrade retrieval quality. This is exactly where layout-aware parsing becomes essential.

The Shift Towards Layout‑Aware Parsing

Layout‑aware parsing treats a document as a visual artifact, not just a text file.

Instead of extracting raw strings, it:

• Detects bounding boxes for semantic blocks
• Classifies elements (Title, NarrativeText, ListItem, Table, etc.)
• Preserves spatial relationships and hierarchy
• Outputs structured data enriched with metadata

This metadata: page numbers, element types, and coordinates, is invaluable for:

• Explainable retrieval
• Debugging
• Evaluation
• High‑quality chunking

For RAG pipelines, layout awareness is not an optimization. It is a prerequisite. The question then becomes how to implement it effectively at scale.

Implementing Layout-Aware Parsing in Practice

Once layout-aware parsing becomes part of the pipeline, the next question is how to implement it reliably at scale. Modern RAG systems rely on a combination of rule-based parsing, machine learning models, and vision-language approaches, depending on document complexity and performance constraints.

Layout-Aware Parsing Tools

Several mature tools now support layout-aware extraction and are widely used in production RAG pipelines:

• Unstructured: A production-ready library that combines heuristics and ML models to classify elements such as titles, narrative text, lists, and tables. It also preserves bounding boxes and metadata, making it well-suited for downstream chunking and explainable retrieval.
• PDFPlumber (layout mode): Provides precise block-level and character-level coordinates. It offers fine-grained control and is often used when teams need custom reading-order logic or debugging visibility.
• LayoutParser/Detectron2/LayoutLMv3: ML-based approaches trained on document layout datasets. These tools perform well on highly variable or noisy formats where rule-based parsers struggle.

Unlike basic extractors, these tools output structured elements instead of raw text, which is critical for reliable retrieval.

Vision‑Language Models: “Seeing” the Document

An emerging approach uses vision-language models (VLMs) that process documents as images rather than plain text. Models such as LayoutLMv3, Donut, GPT-4.1 Vision, Llama 3 Vision, and Qwen2-VL reason over layout, tables, and mixed content using visual cues. These models are especially effective for scanned PDFs, complex tables, and documents with irregular layouts.

In practice, many teams adopt hybrid pipelines—using layout-aware parsers for efficiency and VLMs selectively for the most complex pages. Even with strong parsing models, reconstructing reading flow remains a separate challenge.

Reconstructing Reading Flow in Multi‑Column Layouts

Correctly reconstructing reading flow in multi-column documents requires understanding how text is arranged on the page. Instead of relying on raw text order, the process typically involves:

• Clustering text blocks using spatial coordinates to group related content
• Identifying column regions based on page layout and alignment
• Ordering blocks top to bottom within each column to preserve natural reading flow
• Merging columns in the correct sequence to reconstruct the full narrative

When this step is skipped or done poorly, even high-quality text extraction fails. For RAG systems, accurate reading order is essential to preserve meaning and ensure reliable retrieval. Once the reading order is correct, the next challenge is deciding how that content should be chunked for retrieval.

Intelligent Chunking for RAG Pipelines

Chunking determines how parsed content is broken into retrievable units, and it has a direct impact on retrieval accuracy. Many RAG pipelines begin with baseline chunking because it is easy to implement and works reasonably well for simple text.

Baseline Chunking

Most systems rely on:

• Fixed-size chunks, defined by characters or tokens
• Sliding windows with overlap to reduce context loss

This approach is predictable and fast. However, it is blind to document structure. Chunks may cut through sentences, merge unrelated topics, or separate headings from their content.

What Makes Chunking “Intelligent”?

Intelligent chunking uses document structure and metadata to preserve meaning. Instead of arbitrary splits, it:

• Respects titles, sections, and paragraph boundaries
• Avoids splitting semantic units mid-thought
• Removes repeated headers and footers
• Adapts chunk size based on content density
• Enriches chunks with metadata such as page number, section, and element type

In practice, improving chunking often delivers bigger retrieval gains than switching to a larger model. The next section shows how these concepts translate into an end-to-end implementation.

From Theory to Code: A Reference Implementation

The concepts discussed above are not theoretical. They are implemented end-to-end in our workshop, which serves as a reference RAG pipeline for text-rich, multi-column PDFs.

1. Layout-Aware Parsing (Unstructured, hi_res + fallback)

This is where the pipeline becomes layout-aware instead of just “PDF-to-text”. The script uses Unstructured’s partition_pdf() with hi_res (when available) and falls back to fast if something fails.

from unstructured.partition.pdf import partition_pdf

parse_kwargs = dict(
filename=str(PDF_PATH),
strategy=PARSING_STRATEGY, # "hi_res" preferred
infer_table_structure=True,
extract_images_in_pdf=False,
)

if PARSING_STRATEGY == "hi_res":
parse_kwargs["pdf_hi_res_max_pages"] = HI_RES_MAX_PAGES

try:
elements = partition_pdf(**parse_kwargs)
except Exception as e:
print("⚠️ hi_res parsing failed. Falling back to strategy='fast'.")
elements = partition_pdf(
filename=str(PDF_PATH),
strategy="fast",
infer_table_structure=False,
extract_images_in_pdf=False,
)

What you get back is not raw text. You get typed elements (Title, NarrativeText, etc.) with metadata like page numbers, gold for debugging and evaluation.

2. Parsed Elements → RAG Documents (preserving metadata)

The key here is that we do not throw away structure. We transform each parsed element into a LangChain Document and keep page_number + element_type.

from langchain_core.documents import Document

docs = []
for el in elements:
text = (getattr(el, "text", "") or "").strip()
if not text:
continue

page_number = None
if hasattr(el, "metadata") and el.metadata is not None:
page_number = getattr(el.metadata, "page_number", None)

docs.append(
Document(
page_content=text,
metadata={
"source": PDF_PATH.name,
"page_number": page_number,
"element_type": type(el).__name__,
},
)
)

This design makes retrieval explainable (“why did the model answer that?”) because every chunk can be traced back to a page and element type.

3. Baseline Chunking + Diagnostics (observable chunk quality)

The script uses a baseline splitter, but it doesn’t stop there; it prints diagnostics to expose over-chunking and noise.

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
chunk_size=CHUNK_SIZE,
chunk_overlap=CHUNK_OVERLAP,
length_function=len,
)

chunked_docs = text_splitter.split_documents(docs)

lengths = [len(d.page_content) for d in chunked_docs]
print("min:", min(lengths), "max:", max(lengths), "avg:", sum(lengths)/len(lengths))
print("% very short (<150 chars):", sum(l < 150 for l in lengths) / len(lengths))

That last line is a practical trick: a spike in very short chunks often correlates with headers/footers and layout artifacts.

4. Embeddings + FAISS Index (local, fast, demo-proof)

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

embeddings = OpenAIEmbeddings(model=EMBEDDING_MODEL)
vectorstore = FAISS.from_documents(chunked_docs, embeddings)

vectorstore = FAISS.from_documents(chunked_docs, embeddings)
print("✅ FAISS index created in memory.")

vectorstore.save_local(folder_path=str(FAISS_DIR))
print(f"✅ FAISS index saved to {FAISS_DIR}")

vectorstore = FAISS.load_local(
    str(FAISS_DIR), embeddings, allow_dangerous_deserialization=True
)
print("✅ FAISS index reloaded")

While FAISS is an excellent choice for local experimentation, workshops, and reproducible demos due to its speed and simplicity, production-grade RAG systems often rely on other vector stores depending on scale and operational needs. Common alternatives include Chroma for lightweight prototyping, Milvus and Weaviate for large-scale and metadata-rich retrieval, Qdrant for efficient vector search with strong filtering capabilities, and managed services like Pinecone when low-latency, zero-ops deployments are required. The choice of vector store should be driven by data volume, metadata complexity, and infrastructure constraints rather than embedding model selection.

5. Retrieval First, Then Generation (strict grounding)

A common mistake is to evaluate only the final answer. The script explicitly validates retrieval results (page + type), then runs the full RAG loop with a strict grounding prompt.

user_query = "What ingredients are listed for Avocado Maki?"
k = 4

results = vectorstore.similarity_search(user_query, k=k)

print(f"Top {k} results for query: {user_query!r}\n")
for i, doc in enumerate(results, start=1):
    print(f"--- Result {i} ---")
    print("Page:", doc.metadata.get("page_number"), "| type:", doc.metadata.get("element_type"))
    print(doc.page_content[:600], "\n")

llm = ChatOpenAI(model=CHAT_MODEL, temperature=0.1)
retriever = vectorstore.as_retriever(search_kwargs={"k": k})

prompt = (
    "You are answering strictly from the provided context. "
    "If the answer is not in the context, say you don't know.\n\n"
    "Question: {question}\n\n"
    "Context:\n{context}\n"
)

def rag_answer(question: str):
    docs_ = retriever.invoke(question)
    context = "\n\n".join(
        [f"[p.{d.metadata.get('page_number')}] {d.page_content}" for d in docs_]
    )
    msg = prompt.format(question=question, context=context)
    answer_ = llm.invoke(msg).content
    pages_ = sorted({d.metadata.get("page_number") for d in docs_ if d.metadata.get("page_number") is not None})
    return answer_, pages_, docs_

answer, pages, src_docs = rag_answer(user_query)
print("Answer:\n", answer)
print("\nSource pages:", pages)

This is the simplest, most effective anti-hallucination move: force grounding and surface provenance.

6. Parsing Noise Detection (headers/footers heuristics)

Even layout-aware parsing can include repeated page furniture. The script includes a lightweight heuristic to flag it.

import re
from collections import Counter

def looks_like_header_or_footer(text: str) -> bool:
    lines = [l.strip() for l in text.splitlines() if l.strip()]
    if not lines:
        return False
    first = lines[0].lower()
    last = lines[-1].lower()
    return any(p in first or p in last for p in ["page", "confidential", "draft", "www.", "manual"])

def normalized_lines(text: str):
    return [re.sub(r"\s+", " ", l.strip().lower()) for l in text.splitlines() if l.strip()]

This is intentionally simple: fast signal beats slow perfection when you’re debugging a pipeline.

7. Retrieval Evaluation: Hit@k and MRR (quant, not vibes)

How do you prove that layout-aware parsing is superior? You must measure the retrieval phase using Information Retrieval (IR) metrics. In our workshop, we implemented an evaluation script to calculate:

• Hit@k: The probability that the correct context is within the top k results.
• MRR (Mean Reciprocal Rank): How high the correct answer ranks in the results.

def eval_retrieval(test_set, k=4):
hits = 0
rr_sum = 0.0

for item in test_set:
q = item["q"]
exp = item["expected_page"]
retrieved = vectorstore.similarity_search(q, k=k)
pages = [d.metadata.get("page_number") for d in retrieved]

hit = exp in pages
hits += int(hit)

rr = 0.0
for rank, p in enumerate(pages, start=1):
if p == exp:
rr = 1.0 / rank
break
rr_sum += rr

hit_rate = hits / len(test_set)
mrr = rr_sum / len(test_set)
return hit_rate, mrr

Evaluating Parsing Before It Breaks Retrieval

Evaluation is often overlooked in RAG pipelines, yet it is essential. Many teams only evaluate final answers, even though most failures originate earlier during parsing and retrieval. Catching issues at this stage prevents wasted effort downstream and makes system behavior easier to explain and improve.

Qualitative Checks

Start with simple human inspection:

• Do paragraphs read naturally from start to finish?
• Are sentences complete and coherent?
• Are columns clearly separated and ordered correctly?
• Is document hierarchy—titles, sections, and lists—preserved?

These checks often reveal problems faster than automated metrics.

Noise Detection

Lightweight heuristics can quickly flag common parsing artifacts:

• Repeated headers or footers
• Page numbers embedded in content
• Broken or partial OCR fragments

Removing this noise significantly improves chunk quality.

Retrieval-Level Metrics

Once documents are indexed, retrieval quality should be measured directly:

• Hit@k: Is the correct context retrieved within the top k results?
• MRR (Mean Reciprocal Rank): How early does the correct result appear?

These metrics are easy to interpret and immediately actionable.

Building Reliable RAG Systems Starts With Parsing

Modern RAG systems succeed or fail based on what they see. Layout-aware parsing, correct reading order, intelligent chunking, and systematic evaluation form the foundation of reliable retrieval. When these steps are done well, models hallucinate less, retrieval becomes explainable, and answers remain grounded in source documents.

If you take one action after reading this article, run a layout-aware parser on your most problematic PDF and inspect the output. The improvement is often immediate.

If you’re exploring how to build or improve a production-grade RAG system for complex documents, the Omdena team can help. Book an exploration call with Omdena to discuss your use case and see how layout-aware, human-centered AI systems can deliver reliable results at scale.