[Step 1-A-3/4] Chunking, Token Limits & Surviving Timeouts (Developer Edition)

📌 Expert Workflows · Step 1: AI PDF Summarizer · Edition A (Developer) · Part 3 of 4 (Chunking, Token Limits & Surviving)

🎯 What You’re Actually Building Here

You have a configured API connection from part 1 and extracted text from part 2. Verify both with your own provider account and authorised test PDF before continuing. The obvious next move is to paste all of it into one request and ask for a summary. A short document may fit in one request, while a longer one can exceed the selected model’s context budget or lose important detail. The threshold varies by model, prompt, language, and output allowance.

This tutorial builds a paragraph-aware chunker, a map-reduce summarization pattern, and bounded retry handling. Capacity still depends on the model context window, provider limits, document structure, tokenisation, available time, and budget.

Project context

Scope. The timeout and failure examples below are controlled troubleshooting examples, not claims from a production run. Provider behaviour, SDK defaults, latency, and retry guidance vary by service and version.

🛠️ What to Have Ready

• Parts 1 and 2 complete: working API connection, extract_text() and inspect_pdf()
• A longer test PDF — 20+ pages is ideal for feeling the difference

How to Use This Troubleshooting Tutorial

• 🔴 BEFORE shows an intentionally incomplete or flawed example so readers can reproduce and understand a representative problem.
• 🧠 CAUSE explains why the problem occurs.
• 🟢 AFTER presents the corrected implementation for this lesson.
• ✅ VERIFICATION explains how to confirm that the correction works.

📝 Step 1: Why One Big Request Fails

Every model has a context window — the maximum amount of text it can hold at once, measured in tokens rather than characters (roughly 750 words per 1,000 tokens in English; more tokens for other languages, which is why the same word budget in Korean or Japanese costs noticeably more tokens). You can see exactly how text splits into tokens with OpenAI’s tokenizer guide. If you’re fuzzy on why, how AI models actually work covers tokens properly.

Three separate problems appear when you ignore this:

1. Hard rejection. Exceed the window and the request errors out.
2. Silent degradation. Stay just inside it and quality drops — models reliably use information at the beginning and end of a long context better than material buried in the middle. This isn’t folklore: the research paper “Lost in the Middle” (Liu et al., 2023) measured exactly this U-shaped performance curve across several models. Your summary quietly under-represents the middle of the document — which for a 40-page report is often where the substance lives, so the failure is not just real but actively targets the part you most needed.
3. Timeouts. Long inputs mean long processing, and the default client timeout often expires first.

The fix for all three is the same: send less text per request, more times.

📝 Step 2: A Paragraph-Aware Chunker

Naive slicing (text[0:6000]) cuts sentences in half and destroys the context the model needs. Split on paragraph boundaries instead:

def chunk_text(text: str, max_chars: int = 6000, overlap: int = 200) -> list[str]:
    """Split text into chunks that respect paragraph boundaries."""
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks: list[str] = []
    current = ""

    for para in paragraphs:
        if len(para) > max_chars:
            if current:
                chunks.append(current.strip())
                current = ""
            step = max_chars - overlap
            for i in range(0, len(para), step):
                chunks.append(para[i:i + max_chars])
            continue

        if len(current) + len(para) + 2 <= max_chars:
            current += para + "\n\n"
        else:
            chunks.append(current.strip())
            current = para + "\n\n"

    if current.strip():
        chunks.append(current.strip())

    return chunks

Why max_chars=6000? It’s a deliberately conservative default. Characters aren’t tokens, and the ratio varies by language — the same 6,000 characters can be 1,500 tokens of English or considerably more in Korean, Japanese, or Chinese. Starting conservatively reduces context-limit risk, but it does not guarantee compatibility. Measure tokens with the tokenizer for the selected model and reserve space for prompts and output before increasing the limit.

A character limit is only a safety estimate, not an exact token budget. The real request includes the system message, the user prompt, the chunk text, the requested output, and—depending on the model—reasoning tokens.

Before sending a request, keep this rule in mind:

input tokens + reserved output tokens + safety margin < model context window

The same number of characters can produce different token counts depending on the language, tokenizer, and model. Use the provider’s tokenizer or token-counting tool when exact sizing matters.

The overlap matters only when a single paragraph is longer than a chunk — repeating a couple hundred characters at the seam stops a sentence from being severed mid-thought.

📝 Step 3: Map-Reduce Summarization

A practical map-reduce pattern for documents that exceed a single-request budget:

• Map: summarize each chunk independently
• Reduce: summarize the collected summaries into a final document summary

def summarize_chunk(client, model: str, chunk: str, index: int, total: int) -> str:
    prompt = (
        f"You are summarizing section {index} of {total} from a longer document.\n"
        "Write 3-5 bullet points capturing only the factual content of this section. "
        "Do not add information that is not present. If the section is boilerplate "
        "(headers, page numbers, legal footers), reply with exactly: SKIP\n\n"
        f"---\n{chunk}\n---"
    )
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2,
        max_tokens=500,
    )
    return response.choices[0].message.content.strip()

Three prompt decisions doing real work here. Telling the model it’s seeing section 3 of 12 stops it from writing “In conclusion…” for every chunk. temperature=0.2 keeps summaries factual rather than creative. And the SKIP instruction prevents four bullet points of insight about a page that only contained a header and a page number — filtering those out can reduce noise in the final result, which should be checked on representative documents.

The reduce step:

def reduce_summaries(client, model: str, summaries: list[str]) -> str:
    combined = "\n\n".join(summaries)
    prompt = (
        "Below are section summaries from a single document, in order.\n"
        "Write a unified summary: one short paragraph of overview, then "
        "5-8 bullet points of the most important content. Remove duplicates. "
        "Use only what appears below.\n\n"
        f"---\n{combined}\n---"
    )
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2,
        max_tokens=900,
    )
    return response.choices[0].message.content.strip()

If the combined summaries are too long for one request, chunk and reduce them again. Each additional reduction adds calls, cost, latency, and another opportunity for omission, so test the hierarchy on representative authorised documents rather than assuming unlimited scale.

📋 The Chunk-Size Trade-off

Choosing a chunk size isn’t arbitrary — it’s a trade-off between three competing costs, and getting it wrong shows up as either bad summaries or a slow, expensive run. Keep this as your tuning guide:

The rule this encodes: chunk on natural boundaries (paragraphs), not fixed character counts, because splitting mid-sentence hands the model half an idea and gets you half an answer. The paragraph-aware chunker above does exactly this — it’s slightly more code than a naive character split, and it’s the difference between a summary that reads coherently and one that has gaps where ideas were severed.

⚠️ Troubleshooting — A request times out mid-run

🔴 BEFORE

openai.APITimeoutError: Request timed out.

This controlled failure example shows what happens when an API request times out after earlier chunks have completed. The exact chunk number, timing, error class, billing outcome, and retry behaviour depend on the provider and installed SDK version.

🤔 Cause

Timeout and automatic-retry defaults depend on the installed client version and provider. Check the current SDK documentation and set explicit values appropriate to the task rather than relying on an assumed default.

This tutorial sets `timeout=90.0` and `max_retries=0` deliberately so that the example’s own retry loop controls the behaviour. Other OpenAI-compatible providers may use different defaults.

The second—and more expensive—mistake is the lack of partial-result handling. Without catching the exception inside the loop, one failed request can discard the successful results that came before it.

🟢 AFTER

Use an explicit timeout, bounded backoff, and per-chunk error reporting:

import random
import time
from openai import (
    OpenAI,
    APIConnectionError, 
    APIStatusError, 
    RateLimitError,
)

client = OpenAI(
    api_key=API_KEY,
    base_url=BASE_URL,
    timeout=90.0,
    max_retries=0,   # we handle retries ourselves
)

def with_retries(fn, attempts: int = 4, base_delay: float = 2.0):
    for attempt in range(1, attempts + 1):
        try:
            return fn()
        except (APIConnectionError, RateLimitError) as exc:
            if attempt == attempts:
                raise
            error_name = type(exc).__name__

        except APIStatusError as exc:
            retryable_status = (
                exc.status_code in {408, 409, 429}
                or exc.status_code >= 500
            )

            if not retryable_status or attempt == attempts:
                raise

            error_name = type(exc).__name__
        delay = (
            base_delay * (2 ** (attempt - 1)) 
            + random.uniform(0, 1)
        )
        print(
            f"  attempt {attempt} failed "
            f"({error_name}) — retrying in {delay:.1f}s"
        )
        time.sleep(delay)

Then make the pipeline resilient rather than all-or-nothing:

def summarize_document(client, model: str, text: str) -> str:
    chunks = chunk_text(text)
    print(f"{len(chunks)} chunks to process")

    summaries, failed = [], []

    for i, chunk in enumerate(chunks, start=1):
        print(f"[{i}/{len(chunks)}] summarizing…")
        try:
            result = with_retries(
                lambda: summarize_chunk(client, model, chunk, i, len(chunks))
            )
            if result != "SKIP":
                summaries.append(result)
        except Exception as exc:
            print(f"  chunk {i} failed permanently: {exc}")
            failed.append(i)

    if failed:
        print(f"warning: {len(failed)} chunk(s) missing from the summary: {failed}")

    if not summaries:
        raise SystemExit("No chunks summarized successfully.")

    return reduce_summaries(client, model, summaries)

Exponential backoff with jitter isn’t ceremony: retrying instantly hits the same congestion, and retrying on a fixed schedule synchronizes your retries with everyone else’s. Doubling the delay plus a random fraction spreads the load out.

Verification boundary: catching an exception inside the loop preserves successful summaries only while this Python process remains alive. It does not create a durable checkpoint. To resume after a crash or restart, write each completed chunk result and its source identifier to approved storage, then validate those checkpoints before reusing them. Do not log document text, credentials, or confidential provider responses.

⚠️ What Still Needs Doing

A pipeline that succeeds on average is not the same as a pipeline you can trust. Two habits separate the two, and both cost almost nothing to add now.

Report what’s missing. That failed list is the difference between a summary with a silent hole in it and a summary that says “sections 7 and 12 could not be processed.” A summary that quietly omits a chapter is worse than an error, because it looks complete.

Watch your call count. Map-reduce turns one document into N+1 API calls. A long PDF can require many map calls plus one or more reduce calls. Print the chunk count, estimate token usage and provider cost, and confirm current account limits before processing. Quotas, pricing, and rate limits vary by provider, model, account, and date.

✅ What You Have Now

A chunker that respects paragraph boundaries, a map-reduce pipeline that can handle very long documents when the reduce stage is also kept within the model’s context window, retry logic with exponential backoff and jitter, per-chunk failure isolation so one bad response can’t destroy a run, and honest reporting of gaps. Test it with authorised documents of different lengths and compare every summary with representative source passages. Successful execution does not prove that every section was represented accurately.

💡 An Optional Upgrade

Cache chunk summaries to disk keyed by a hash of the chunk text. Re-running the pipeline after changing only the reduce prompt then costs one API call instead of eighty — the single biggest quality-of-life improvement while you’re iterating on prompts.

❓ This Week’s Task

Run the pipeline on your longest test PDF and note the chunk count before it starts. Then deliberately break something: set timeout=0.001 and confirm you see retry messages and a clean permanent-failure report rather than a raw traceback. Watching your own error handling work is how you learn to trust it.

🔗 Next in This Series

[Step 1-A-4/4] Structured JSON Output & Shipping the Tool, where we stop printing loose prose and start returning parseable data — plus the JSON parsing failure that can occur when a model adds prose or returns malformed data, and the argparse CLI that turns this into a tool you actually use.

✍️ About the author & how this was made

Retry and cost boundary

Retries can duplicate billable requests or actions. Use explicit timeouts, capped exponential backoff, idempotency where available, cost limits, progress checkpoints, and logs that do not contain document text or secrets. Verify the reduced summary against representative source passages.

Verified Chunking Evidence

Tested: 30 August 2026 on Windows 10 with local Python. These deterministic checks used synthetic text and made no live AI-provider request.

Download the secret-free verified chunking package (ZIP)

SHA-256: B75B3FBA7F66003FA9054080F4A520D48189BFE880882860CE04F949CD367E32

Verification boundary: These checks verify local control flow and deterministic transformations. They do not prove provider availability, model quality, exact token counts for a selected model, request cost, latency, SDK retry behaviour, or production reliability. Test authorised representative documents and the selected provider’s current documentation before deployment.