Executive Summary
This document covers the architecture, failure modes, and evaluation strategies for building a robust PDF-to-Markdown pipeline that handles both native (text-based) PDFs and scanned (image-based) documents using vision AI models.
Architecture Overview
The pipeline consists of five major stages, with conditional branching based on PDF type detection. Here's the high-level flow:
Pipeline Architecture Diagram
┌─────────────────────────────────────────────────────────────────────────────┐
│ PDF INPUT │
│ (digital or scanned) │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ STAGE 1: PDF TYPE DETECTION │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Extract text │───▶│ Text present? │───▶│ Digital PDF path │ │
│ │ layer metadata │ │ (>100 chars/page)│ │ │ │
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
│ │ No │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Scanned PDF path │ │
│ │ (OCR required) │ │
│ └──────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌─────────────────────────┴─────────────────────────┐
▼ ▼
┌──────────────────────────┐ ┌──────────────────────────┐
│ DIGITAL PDF BRANCH │ │ SCANNED PDF BRANCH │
│ │ │ │
│ ┌────────────────────┐ │ │ ┌────────────────────┐ │
│ │ Text extraction │ │ │ │ Page rasterization │ │
│ │ (pdfplumber/PyMuPDF) │ │ │ (300 DPI PNG) │ │
│ └────────────────────┘ │ │ └────────────────────┘ │
│ │ │ │ │ │
│ ▼ │ │ ▼ │
│ ┌────────────────────┐ │ │ ┌────────────────────┐ │
│ │ Layout analysis │ │ │ │ Vision AI OCR │ │
│ │ (positions, fonts) │ │ │ │ (GPT-4V / Claude) │ │
│ └────────────────────┘ │ │ └────────────────────┘ │
│ │ │ │ │ │
│ ▼ │ │ ▼ │
│ ┌────────────────────┐ │ │ ┌────────────────────┐ │
│ │ Structure mapping │ │ │ │ Confidence scoring │ │
│ └────────────────────┘ │ │ └────────────────────┘ │
└──────────────────────────┘ └──────────────────────────┘
│ │
└─────────────────────────┬─────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ STAGE 3: LAYOUT SEGMENTATION │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Block detection │───▶│ Block typing │───▶│ Bounding boxes │ │
│ │ (text regions) │ │ (H1/H2/P/List) │ │ extraction │ │
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ STAGE 4: READING ORDER INFERENCE │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Column detection │───▶│ Top-to-bottom │───▶│ Multi-column │ │
│ │ │ │ left-to-right │ │ handling │ │
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ STAGE 5: MARKDOWN SYNTHESIS │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Block → Markdown │───▶│ Whitespace │───▶│ Output modes: │ │
│ │ mapping │ │ normalization │ │ • MD only │ │
│ └──────────────────┘ └──────────────────┘ │ • MD + JSON │ │
│ │ • Per-page MD │ │
│ └──────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘Stage Details
PDF Type Detection
Extract the text layer and count characters per page. If most pages have <100 extractable characters, route to the OCR branch. This avoids wasting compute on native PDFs while ensuring scanned docs get proper treatment.
Text/OCR Extraction
Digital path: Use pdfplumber or PyMuPDF to extract text with font metadata and positional coordinates.
Scanned path: Rasterize pages to 300 DPI PNG, then pass to vision AI (GPT-4V, Claude Vision, or Azure OpenAI) for text extraction with confidence scores.
Layout Segmentation
Cluster text runs into blocks (paragraphs, headings, lists, tables, captions). Use font size ratios and whitespace gaps to identify block types. Output bounding boxes for downstream QA.
Reading Order Inference
Handle multi-column layouts by detecting column boundaries. Sort blocks top-to-bottom within each column, then left-to-right across columns. This is critical for legal documents, academic papers, and newspapers.
Markdown Synthesis
Map block types to Markdown: H1/H2/H3 for headings (based on font size hierarchy), bullet/numbered lists for list items, pipe tables for detected tables. Normalize whitespace and fix hyphenation artifacts.
Failure Modes & Edge Cases
Production pipelines encounter predictable failure patterns. Here are the most common issues and mitigation strategies:
1. Multi-Column Reading Order Errors
Symptom: Text from different columns gets interleaved, producing nonsensical output like "The company reported Q3 earnings of $4.2 billion in revenue, marking growth of year-over-year..."
Root cause: Column detection failed, treating the entire page as single-column.
Mitigation:
- • Detect vertical whitespace gaps >30px as column boundaries
- • Use x-coordinate clustering to identify column centers
- • Fall back to vision AI for complex layouts
2. Table Structure Collapse
Symptom: Tables become plain text with lost column alignment. "Name: John Age: 32 City: NYC Name: Jane Age: 28..."
Root cause: Table detection missed due to borderless tables or inconsistent cell spacing.
Mitigation:
- • Use vertical alignment patterns to detect table structures
- • Implement heuristics for repeated horizontal patterns
- • Vision AI excels at table detection—route complex tables to OCR path
3. OCR Confidence Degradation on Low-Quality Scans
Symptom: Character substitutions like "rn" → "m", "l" → "1", "O" → "0". Words become garbled.
Root cause: Low DPI scans, skewed pages, or faded ink reduce OCR accuracy.
Mitigation:
- • Pre-process with de-skew and de-noise filters
- • Upscale low-DPI images before OCR
- • Use confidence thresholds to flag suspect regions
- • Implement spell-check post-processing for domain vocabularies
4. Heading Hierarchy Misdetection
Symptom: All headings become H1, or subheadings get promoted to main headings, breaking document structure.
Root cause: Font size detection failed, or document uses visual styling (bold/color) instead of size for hierarchy.
Mitigation:
- • Build relative font size ratios per document (not absolute thresholds)
- • Consider font weight, color, and position in hierarchy detection
- • Use document outline/bookmarks when available as ground truth
5. Hyphenation and Line Break Artifacts
Symptom: Words broken across lines remain split: "This docu- ment describes the imple- mentation..."
Root cause: PDF preserves visual line breaks, and naive extraction keeps them.
Mitigation:
- • Detect hyphen + newline pattern and rejoin words
- • Use dictionary lookup to validate joined words
- • Preserve intentional hyphens (compound words) by checking both halves
Evaluation Checklist
Use this checklist to validate pipeline output quality before deploying to production:
Quality Assurance Checklist
Text Accuracy
- Character error rate (CER) < 2% for digital PDFs
- CER < 5% for scanned PDFs (clean scans)
- No hyphenation artifacts in continuous text
- Special characters (€, ©, ™) preserved correctly
Structure Preservation
- Heading hierarchy matches source document
- Lists (bullet and numbered) preserved with correct nesting
- Tables converted with correct column count and alignment
- Multi-column text in correct reading order
Layout Fidelity
- Page breaks indicated (for per-page output mode)
- Headers/footers excluded or marked separately
- Figure captions associated with correct images
- Footnotes/endnotes linked or inline
RAG Readiness
- Output parses as valid Markdown
- Chunk boundaries align with semantic sections
- Metadata (page numbers, source file) included
- Token count reduced 50%+ vs raw PDF text dump
Internal Benchmarks
We continuously measure pipeline performance across diverse document types. Here are our current metrics:
Performance Benchmarks (Q1 2025)
| Metric | Digital PDF | Scanned (Clean) | Scanned (Low-Q) |
|---|---|---|---|
| Character Error Rate | 0.3% | 2.1% | 6.8% |
| Word Error Rate | 0.8% | 3.4% | 9.2% |
| Table Detection F1 | 94.2% | 89.7% | 78.3% |
| Heading Detection F1 | 96.1% | 91.4% | 87.6% |
| Reading Order Accuracy | 98.7% | 95.2% | 92.1% |
| Avg. Processing Time (20p) | 3.2s | 18.5s | 24.1s |
Benchmark dataset: 500 documents across legal, academic, financial, and technical domains. "Low-Q" = scans <200 DPI or with visible degradation.
Chunking Strategies for RAG
The way you chunk converted Markdown significantly impacts retrieval quality. Here are proven strategies:
Section-Based Chunking
Split on H1/H2 headings. Each chunk is a self-contained section with its heading as context.
## Legal Framework This section covers the regulatory requirements for... --- chunk_id: doc1_section_2 heading: "Legal Framework" page_range: [4, 7] token_count: 412
Best for: Legal docs, manuals, structured reports
Sliding Window + Overlap
Fixed token windows (e.g., 500 tokens) with 20% overlap to preserve context across boundaries.
# Chunk 1 (tokens 0-500) ...end of paragraph about costs. # Chunk 2 (tokens 400-900) ...about costs. The implementation requires careful planning... # 100-token overlap preserves context
Best for: Narrative content, books, long-form articles
Semantic Paragraph Chunking
Group 2-4 consecutive paragraphs that discuss the same topic, detected via embedding similarity.
[Paragraphs about data privacy] Companies must ensure... GDPR compliance requires... Data retention policies should... --- topic_cluster: "data_privacy" similarity_score: 0.89
Best for: Research papers, news articles, Q&A content
Hybrid: Parent-Child Chunks
Store both large parent chunks (for context) and smaller child chunks (for precision retrieval).
# Parent chunk (full section) ## Implementation Guide [2000 tokens of content...] # Child chunks (paragraphs) ├── child_1: "Step 1: Configure..." ├── child_2: "Step 2: Deploy..." └── child_3: "Step 3: Validate..."
Best for: Technical docs, FAQs, hierarchical content
Output Modes
Markdown Only (Default)
Clean .md output with headings, lists, tables, and prose. Fastest processing, smallest output size. Use for direct RAG ingestion where you don't need positional data.
Markdown + JSON Annotations
Markdown content plus a JSON sidecar with bounding boxes, confidence scores, and block metadata. Use for QA workflows, debugging, or when you need to map text back to visual positions.
{
"blocks": [
{
"type": "heading",
"level": 1,
"text": "Executive Summary",
"bbox": [72, 120, 540, 145],
"confidence": 0.98,
"page": 1
}
]
}Per-Page Markdown
One .md file per page with page number metadata. Use for page-level chunking strategies or when you need to cite specific pages in RAG responses.
Ready to Build Your Pipeline?
Try our API for instant PDF to Markdown conversion with OCR support. Production-ready infrastructure, no setup required.