[Step 1-A-2/4] Extract Text From PDFs — Detect Scans and Encoding Problems (Developer Edition)

📌Expert Workflows · Step 1: AI PDF Summarizer · Edition A (Developer) · Part 2 of 4 (Extracting Text From Any PDF)

🎯 The Reason This Matters

Inpart 1you proved your machine can talk to an AI model. Now we get the text out of the PDF — and this is the part that surprises most developers. A PDF isn’t a document format so much as a set of drawing instructions — thePDF specification (ISO 32000)describes a page as positioned glyphs and graphics, not as a stream of readable words — and “get the text” ranges from one line of code to genuinely impossible depending on how the file was made. A text-based PDF stores the characters; a scanned PDF stores apictureof characters, with no text layer at all, which is why the same line of code returns a full document for one file and an empty string for another.

By the end of this post you’ll have a reusable extraction function, a validator that tells youimmediatelywhether a PDF is usable, and fixes for the two failures that will otherwise send you in circles: text that comes back empty, and text that comes back garbled.

Project context

Validation should precede summarisation: test both a text-based PDF and a scanned or image-only PDF so an empty extraction is treated as a limitation, not usable input.

🛠️ Before You Start

• Everything from part 1 (project folder,.venv, working.env, verified API connection)
• pypdfinstalled (pip install pypdf)
• Two test PDFs: one text-based (exported from a word processor), one scanned (a photo or scan of paper)

How to Use This Troubleshooting Tutorial

• 🔴 BEFORE shows an intentionally incomplete or flawed example so readers can reproduce and understand a representative problem.
• 🧠 CAUSE explains why the problem occurs.
• 🟢 AFTER presents the corrected implementation for this lesson.
• ✅ VERIFICATION explains how to confirm that the correction works.

📝 Step 1: The Extraction Function

We’ll usepypdf, whoseofficial documentationcovers the extraction edge cases in depth. Add this to a new file,pdf_reader.py:

from pathlib import Path
from pypdf import PdfReader

def extract_text(pdf_path: str) -> str:
    """Return all text from a PDF, page by page."""
    path = Path(pdf_path).expanduser().resolve()

    if not path.exists():
        raise FileNotFoundError(f"No PDF at: {path}")

    reader = PdfReader(str(path))
    pages = []

    for page in reader.pages:
        pages.append(page.extract_text() or "")

    return "\n\n".join(pages)

if __name__ == "__main__":
    text = extract_text("sample.pdf")
    print(f"characters extracted: {len(text)}")
    print(text[:500])

Two details worth noticing.page.extract_text()returnsNonefor pages with no extractable text, soor ""prevents aTypeErrorfrom taking down a 200-page run at page 137. And.expanduser().resolve()turns~/Documents/file.pdfand relative paths into one absolute path, so the error message tells you exactly where Python looked.

Run it:

python pdf_reader.py

On a text-based PDF you’ll see a character count and the first 500 characters. On a scanned one you’ll seecharacters extracted: 0— which is where most tutorials abandon you.

📋 The Three PDF Types You’ll Meet

Every PDF you feed this tool falls into one of three buckets, and knowing which one you’re holding tells you immediately whether extraction will work — before you waste an AI call on empty text:

The practical rule this encodes:always check the character count before sending text to the AI.An empty extraction that silently becomes an AI request produces a confident summary ofnothing— the model doesn’t know the text is missing, so it invents plausible content, and you’ve now got a fabricated summary that looks real. The validator you built above is what stops that, and it’s why the character count is the most important number in the whole extraction step.

Understand the symptom

The next result is intentional. Place a scanned or image-only PDF in the project folder as `sample.pdf`, then run:

python pdf_reader.py

If the output says `characters extracted: 0`, the tutorial is working as designed. This is not a crash: it demonstrates what happens when a PDF contains images but no usable text layer. We will diagnose and fix this result below.

⚠️ Troubleshooting — The PDF that extracted nothing

🔴 BEFORE

characters extracted: 0

No exception. No warning.pypdfdid its job correctly and the pipeline continued happily — sending an empty string to the AI, which returned a confident summary of nothing at all.

🤔 Cause

The file is a scan: an image wrapped in a PDF container. A text extractor can correctly return zero characters because no searchable text layer exists. The workflow should stop, report the limitation, and route the document to an approved OCR process rather than asking a model to summarise missing text.

🟢 AFTER

Validate before you spend an API call:

MIN_CHARS_PER_PAGE = 50

def inspect_pdf(pdf_path: str) -> dict:
    path = Path(pdf_path).expanduser().resolve()
    reader = PdfReader(str(path))

    per_page = [len((p.extract_text() or "").strip()) for p in reader.pages]
    total = sum(per_page)
    empty_pages = [i + 1 for i, n in enumerate(per_page) if n < MIN_CHARS_PER_PAGE]

    return {
        "pages": len(per_page),
        "total_chars": total,
        "empty_pages": empty_pages,
        "likely_scanned": total < MIN_CHARS_PER_PAGE * len(per_page) / 2,
    }

Then refuse to continue on a file you can’t read:

info = inspect_pdf("sample.pdf")
if info["likely_scanned"]:
    raise SystemExit(
        f"This PDF may be scanned or may contain too little extractable text. "
        f"Inspect the file or run OCR before continuing "
        f"({info['total_chars']} chars across {info['pages']} pages)."
    )

For scanned files, the fix is OCR —ocrmypdfis the pragmatic option because it writes anewPDF with a real text layer, meaning the rest of your pipeline needs zero changes:

ocrmypdf input_scanned.pdf output_with_text.pdf

OCRmyPDF is a separate command-line tool. Install it according to the official instructions for your operating system before running this command. It adds a searchable text layer to scanned PDFs.

💡 Teaching note:the cost of this one wasn’t the debugging, it was the trust. A summary of an empty document looked completely plausible — which is exactly the failure mode covered inAI Hallucination & Ethics. Garbage in doesn’t produce an error; it produces confident garbage out. Validate inputs, always.

📝 Step 2: Saving the Extracted Text

You’ll want the raw text on disk while developing, so you’re not re-parsing a large PDF on every run:

def save_text(text: str, out_path: str = "extracted.txt") -> None:
    Path(out_path).write_text(text, encoding="utf-8")
    print(f"saved {len(text)} characters to {out_path}")

Thatencoding="utf-8"is not optional, and the next Troubleshooting notes explains why.

⚠️ Troubleshooting — Question marks, broken glyphs, and a Windows crash

🔴 BEFORE

Two symptoms, same root area. On Windows, printing extracted text to the console:

UnicodeEncodeError: 'cp949' codec can't encode character '\uc548' in position 0

And in the extracted text itself, non-English characters arriving as?????or as visually scrambled glyphs.

🤔 Cause

These are two different problems that get lumped together as “encoding issues.”

The crash is anoutputproblem: Python produced perfectly valid Unicode, but the Windows console’s legacy code page can’t represent those characters. Nothing is wrong with your data — only with where you tried to display it.

The scrambled glyphs are aninputproblem: some PDFs embed fonts with a custom character map and noToUnicodetable. The file knows which shape todrawfor each code, but not which character that shapeis. Copy-pasting from the PDF in a viewer produces the same nonsense, which is the quickest way to confirm the file is at fault rather than your code.

🟢 AFTER

For the output problem — write to files with an explicit encoding, and force the console when you need it:

# Always explicit, never platform-dependent
Path("extracted.txt").write_text(text, encoding="utf-8")

# Windows PowerShell, current session
$env:PYTHONIOENCODING = "utf-8"

# Windows Command Prompt, current session
set PYTHONIOENCODING=utf-8

For the input problem — detect it rather than silently summarizing nonsense:

def looks_garbled(text: str, sample: int = 2000) -> bool:
    """Rough check: lots of replacement/undefined characters in a sample."""
    chunk = text[:sample]
    if not chunk:
        return False
    bad = sum(1 for ch in chunk if ch in "\ufffd" or ord(ch) == 0)
    return bad / len(chunk) > 0.05

If it returnsTrue, the practical fix is the same as for scans: run the file through OCR to rebuild a clean text layer.

Verification note: This controlled example represents a common PDF text-extraction failure. Use the inspection step to determine whether the PDF contains an extractable text layer before choosing an approved OCR process.

⚠️ Loose Ends Worth Tying

Extraction that “works on my test file” is the most common source of silent failures in document pipelines. Real-world PDFs include scans, mixed-language documents, multi-column layouts that extract in reading-order chaos, and files where half the pages have a text layer and half don’t. You are not writing a parser for one file; you’re writing a gate that decides whether a file is safe to summarize.

That’s whyinspect_pdf()matters more thanextract_text(). The extractor is ten lines anyone can copy. The validator is the part that keeps a fabricated summary of an empty document out of your output — and later, out of your users’ hands.

✅ What You Have Now

A resolved-path extraction function that survives empty pages, an inspector that reports page count, character totals, empty pages and a likely-scanned flag, UTF-8-safe file output, and a garbled-text detector. Your pipeline can now tell the difference between “this PDF has no text” and “this PDF has text I mangled” — a distinction that will matter in every document project you ever build.

💡 An Optional Upgrade

Runinspect_pdf()across a folder of PDFs and print a small report. You’ll learn more about real-world PDF quality in five minutes than in any article — and you’ll have built the first useful diagnostic tool of this course.

❓ Put It Into Practice

Extract text from both of your test files. Confirm the text-based one returns thousands of characters and the scanned one triggers yourlikely_scannedguard rather than sailing through. If you don’t have a scan handy, photograph a printed page with your phone and save it as a PDF.

🔗 Next in This Series

[Step 1-A-3/4] Chunking, Token Limits & Surviving Timeouts, where we feed that extracted text to the model — and deal with the fact that a 40-page PDF doesn’t fit in one request, plus the retry logic that keeps a long run from dying at page 12.

Written and edited by G. Troy for Life Tech Hack, a beginner-friendly technology site. This tutorial explains how Python PDF text extraction behaves when a document contains a searchable text layer and when it does not. The error output and example results are included to make the debugging process easier to understand. They are teaching examples unless your own test produces the same result. Your output may differ depending on your operating system, Python packages, PDF file, and OCR tools. Always verify the result with your own file before relying on it.

Document handling boundary

Use public, owned, or authorised PDFs. Text extraction can scramble reading order or return nothing; OCR introduces additional errors and privacy considerations. Do not upload confidential, personal, copyrighted, legal, medical, or financial documents to an unapproved service.

Verified PDF Extraction Evidence

Tested locally on August 30, 2026 with Python, pypdf 6.10.0, ReportLab-generated searchable text, and a synthetic blank-page PDF. No private document or external AI service was used.

SHA-256: B44DBAC57845705629DB38D8C0255D9F32683D4B9A3D524C83D90F9D1360723A

Verification boundary: the synthetic test proves that the included functions extract a simple searchable-text PDF, flag an empty page for inspection, write UTF-8 text, and reject a missing path. The character-count heuristic cannot prove that a file is scanned, and this package does not perform OCR. Encrypted, damaged, unusual-font, rights-restricted, or complex-layout PDFs require separate authorized testing.