From a scanned PDF to an answer you can trust

Most “chat with your PDF” demos work right up until the moment the answer matters.

Ask one for a definition and it sounds great. Ask it for a torque figure, a valve clearance, or a part number, and it will still sound great — confidently wrong. The difference between a demo and something you would actually let a customer or an engineer rely on is whether the system is measured and grounded at every step, or just trusting a model to have read carefully.

I build small, real things to find out where the hard parts are. The latest is a set of assistants over decades-old motorcycle service manuals — the kind of document that is hostile to AI in every way: scanned pages, dense specification tables, exploded parts diagrams, and thousands of exact values where being one digit out is the same as being wrong.

Three of them are live:

• virago.edestudio.us — 1994 Yamaha XV250 Virago
• vfr750.edestudio.us — Honda VFR750F
• hondamonkey.edestudio.us — Honda Monkey Z125

Same pipeline behind all three: Docling for extraction, Astra DB hybrid search for retrieval, and a Langflow agent for answering. There is a neat coincidence in that line-up: all three are now IBM. Docling came out of IBM Research in Zurich, and Astra DB and Langflow both joined IBM through its acquisition of DataStax . I did not set out to build an all-IBM stack; I picked the best tool for each layer, and that is where it landed.

If you have read my earlier write-ups on these assistants (the VFR750F , the Monkey Z125 ), this is the part-two. The first version of these sites answered through Cloudflare AI Search , a managed retrieval service, and that version still runs on each site at /classic. The scanned manuals and the scored extraction behind them did not change. What changed is everything after extraction — how the manual is searched and how the answer is assembled — and this rebuild is what the rest of this post is about.

So this post is about why that particular combination is trustworthy on PDFs — and how you would judge whether your documents deserve the same treatment.

The short version: three layers, and not one of them asks you to take the model’s word for it.

Why PDFs break AI ¶

A service manual is the worst case for a document assistant, which makes it the best case to design against.

• The pages are pictures. Old manuals are scans. There is no text to read until something performs optical character recognition (OCR) on every page. Get the OCR wrong and everything downstream inherits the error silently.
• The value is the answer. Nobody opens a workshop manual for vibes. They want “0.07 to 0.12 mm” or “23 N·m” exactly. A summary that is 95% right is 100% useless if the 5% is the number.
• Meaning and symbols both matter. “How do I adjust the valve clearance” is a question about a concept. “0.07 to 0.12 mm” is an exact string. A good answer needs the system to be fluent in both, in the same query.
• People need to verify. A mechanic with their hands in an engine is not going to take a chatbot on faith. The answer has to point back to the page it came from.

Generic “upload your PDF” RAG fails on exactly these points. It runs whatever OCR it ships with and never tells you how good it was. It embeds everything into one kind of search that is good at meaning and bad at exact strings. And it hands back an answer with no way to check it. For casual documents that is fine. For anything a decision or a repair rests on, it is the gap between a toy and a tool.

So the design question is not “can AI read a PDF.” It is: at which steps am I trusting the model, and can I replace that trust with a measurement instead?

Layer 1 — Extraction you can score ¶

Docling is the engine that turns a PDF into structured Markdown: it preserves heading hierarchy, keeps tables as tables, holds specification blocks together, and pulls out the figures. That structure is what makes everything later possible — you cannot retrieve a torque table well if extraction has already smeared it into a paragraph.

But docling is a framework, and the OCR engine inside it is a choice. EasyOCR, RapidOCR, the embedded text layer, a hosted OCR API — they do not perform equally, and the honest truth is you cannot tell which is best by looking. So I do not guess. I score.

I took the 1994 Yamaha XV250 service manual, hand-verified 100 ground-truth values from across it (specifications, clearances, capacities, the things that actually get looked up), and built a harness that runs each extraction backend over the whole manual and checks, with strict exact matching, how many of those 100 values survived. No partial credit. A torque figure is either correct to the digit or it is a miss.

The results were not close:

Two things in that table matter for anyone trusting this with their documents.

First, the spread is real. The version I had originally shipped (EasyOCR) was the worst of the four, 24 points behind the best. If you pick an extractor by reputation or by what came bundled, you are gambling with the one part of the pipeline whose mistakes are invisible. A model will happily reason beautifully over a “23” that should have been “28.” The harness is what turned “this feels fine” into “this one is measurably 24 points better.” I wrote up the full bake-off, including the figure extraction it unlocked, in the local-OCR deep-dive .

Second, the winner gets to change. Today the best engine on this manual was RapidOCR at 85%. Next year a better one will exist — IBM has already shipped Granite-Docling , a purpose-built document-conversion model, since I ran this. Because the harness scores any backend against the same ground truth on the same basis, swapping is a one-line decision backed by a number, not a rewrite and a leap of faith. The extraction code that gets scored is the exact code that produces what ships — so the number you trust is the number you run.

And there is a quieter reason that 85% mattered more to me than a couple of points beyond it. RapidOCR runs locally. A hosted OCR API would have scored a little higher, but the winning local engine reads the page pixels on my own machine: no document leaves it, no per-page API bill, and the exact same pipeline works for a confidential corpus or with no internet at all. For anyone weighing whether to trust AI with sensitive documents, that is the part that should matter — the best-scoring option here is also the one where your files never have to leave your control.

The winning extraction is then made ready for retrieval: every page is given a self-describing header — Source: <manual> | <chapter> | manual page N — and its figure links are rewritten to absolute image URLs. That header is small but it is the seed of everything in Layer 3: it is how an answer later knows which page and which chapter it is standing on.

Why measure instead of trust: OCR errors are the only errors in this whole pipeline that nothing downstream can detect or correct. A wrong retrieval can be reranked away; a wrong answer can be caught by a citation. A wrong digit from OCR just becomes the truth. That is the one place you cannot afford to hope, so it is the one place I insist on a score.

Layer 2 — Storage that searches two ways at once ¶

This is the part that does the heavy lifting, and the part most “PDF chatbot” stacks get half right.

Ordinary RAG turns each chunk of text into a single vector — a numeric fingerprint of its meaning — and searches by semantic similarity. That is genuinely good at conceptual questions. Ask “how do I adjust the valve clearance” and vector search finds the procedure even if the page never uses your exact words.

It is also genuinely bad at the thing a manual is full of: exact symbols. A part number, a torque value, an error code, a precise clearance like “0.07 to 0.12 mm” — these embed poorly. Semantically, “0.07 mm” and “0.09 mm” look almost identical, and “23 N·m” looks like every other small number in the book. The one search that is great at meaning is weak at precisely the values people open the manual for.

This is exactly where version one ran out of road. Cloudflare AI Search got these sites running fast — you point a managed service at your documents and it handles retrieval for you — but it is a single retrieval mode, and the misses clustered on the exact-string questions a service manual lives or dies by. Rebuilding this layer was the whole reason for version two.

So I do not make it choose. Each chunk lands in Astra DB as a hybrid record carrying two indexes:

• $vectorize — the semantic embedding, generated server-side by Astra using NVIDIA’s nv-embedqa-e5-v5 model. The embedding dimension is fixed by the model and there is no embedding API key to hold or rotate; the database does it on write.
• $lexical — a keyword index of the same chunk, the classic exact-term search that has always been good at finding “0.07 mm” because it is looking for the characters, not the meaning.

At query time, both arms run. The vector arm proposes chunks that are about the question; the lexical arm proposes chunks that contain the exact terms. Then a reranker — a model whose only job is to judge relevance — looks at the merged candidate set and reorders it so the genuinely best passages float to the top. Astra exposes this as a single find-and-rerank operation; from the outside it is one query that quietly does the work of three.

Why hybrid is the whole game for technical PDFs

A service manual is, roughly, half prose and half symbols. Any retrieval strategy that is strong on one half is structurally blind to the other:

Pure vector misses the number. Pure keyword misses the intent. The reranker is what lets you combine them without the noise — without it, you would just be pasting two mediocre result lists together and hoping. With it, a conceptual question and a one-digit lookup go through the same endpoint and both come back right.

There is an operational kindness here too. The sync into Astra is incremental: a checksum manifest tracks which pages have changed, so re-running it after a corrected page or a new chapter re-processes only what moved, not the whole corpus. Re-publishing a manual is cheap, which means keeping it correct is cheap, which means it actually stays correct.

Layer 3 — An agent that shows its working ¶

The front of each site is a Langflow agent, served from the page itself, that streams its answer as it thinks and retrieves over the hybrid collection underneath.

The detail that matters to anyone who has to rely on the answer is the one seeded all the way back in Layer 1: citations. Because every chunk still carries its Source: manual | chapter | page N header, the agent’s answers cite the exact manual page they came from, and the citation deep-links there. You are never asked to believe a torque figure on the model’s authority — you are shown which page of the official manual it is reading from, and you can go and look. For a workshop manual that is not a nice-to-have. It is the difference between “the chatbot said 23” and “page 4-11 of the service manual says 23, here it is.”

Each site also keeps the original version-one chat at /classic, built on Cloudflare’s managed AI Search, sitting beside the agentic homepage. Keeping v1 live is partly a fallback and partly an honesty check: it lets me compare the new hybrid agent against the setup it replaced, on the same documents, on the same questions, in public. You can put a precise-spec question to both and see the difference the rebuild actually made.

Why you can actually trust this ¶

Step back from the parts and look at where the trust lives, because that is the only thing that separates this from a confident demo:

1. Extraction is scored against hand-verified truth. The one error class nothing else can catch is the one class I refuse to leave to chance. The shipped extractor was the measured best of four on the reference manual, with a harness that re-scores any future extractor on the same basis — and it runs locally, so the documents never leave the machine.
2. Retrieval covers meaning and exact strings. Hybrid search plus reranking means a conceptual question and a precise-value lookup both land, instead of the system being quietly good at one and quietly blind to the other.
3. Answers cite their page. Nothing rests on “the model said so.” Every answer points back to the official manual page it stands on, and you can open it.

Notice what is not on that list: “we used a really good model.” Models matter, but they are the most replaceable layer and the easiest to over-trust. The durable value is in the boring, verifiable scaffolding around them — and that scaffolding is exactly what a generic upload-your-PDF tool skips.

The pipeline at a glance ¶

  PDF (scanned manual)
        │
        ▼
  ┌─────────────────────┐   scored against 100 hand-verified values
  │ Docling extraction  │   pick the backend that wins (RapidOCR, 85%,
  │ + accuracy harness  │   run locally), not the one that ships by default
  └─────────────────────┘
        │  structure-aware Markdown + per-page "Source: … page N" headers
        ▼
  ┌─────────────────────┐   $vectorize (meaning, server-side NVIDIA embeddings)
  │ Astra DB (hybrid)   │   $lexical   (exact words: torque, clearances, codes)
  │ find-and-rerank     │   reranker fuses both, reorders by true relevance
  └─────────────────────┘   incremental: only changed pages re-sync
        │
        ▼
  ┌─────────────────────┐
  │ Langflow agent      │   streams the answer, cites the exact manual page,
  │ (page-cited)        │   deep-links so a human can verify it
  └─────────────────────┘
        │
        ▼
  Hosted on Cloudflare Pages + R2 (serverless, cheap to keep running)

The economics are deliberately dull. Astra is serverless and the corpora are tiny by database standards; the sync is incremental; the sites run on Cloudflare’s edge. The expensive thinking — deciding which extractor to trust, deciding to store two indexes instead of one — is done once, up front, and then the thing is cheap to operate and cheap to keep correct. That is the order you want those costs in.

What this means for your documents ¶

The motorcycles are incidental. The pattern is not.

If you have a corpus where the values matter — contracts, datasheets, regulatory filings, lab results, anything where “roughly right” is a polite way of saying “wrong” — then the three questions worth asking any AI document tool are exactly the three layers above. How do you know your extraction is correct? Can you find an exact figure as reliably as a concept? Can a human check the answer against the source? If a tool cannot answer those, it is a demo. If it can, it is something you can build on.

You can poke at all three layers right now. Open one of the live assistants and ask it something a search box would choke on — a specific torque value, then a “how do I” procedure, back to back — and watch it handle both and cite the page:

• virago.edestudio.us — Yamaha XV250 Virago
• vfr750.edestudio.us — Honda VFR750F
• hondamonkey.edestudio.us — Honda Monkey Z125

If you ask it a precise specification and it gives you the number and the page, that is all three layers doing their job at once. That is the whole point.

Want the engineering detail — the accuracy harness, the hybrid query internals, the streaming and citation wiring? There is a companion deep-dive: The engineering behind a PDF assistant you can verify .