In April I wrote about the search engine I built over the equipment manuals at the cement plant where I work: 876 PDFs, about 40,000 pages, one search box. It worked. Six people use it.
Then the complaints got specific. Ask for the motor size of a filter fan and you got a VFD manual, a bucket elevator and a mill. The answer read well and cited a page. The page was about the wrong machine.
So this month I stopped adding features and measured it. The number that mattered: for a question worded the way a technician actually talks, the right page made it into the top eight sources 40% of the time.
What was actually wrong, measured on the live database
I had assumptions. I replaced them with queries against the production database before touching any code.
- 44% of the index was duplicates. 72,748 chunks, but only 40,877 distinct texts. The same manual lived on the share as a full binder, as individual tab files and as
_GREYcopies, and all of them were indexed. A top-eight result routinely held four distinct pages. - No chunk knew which machine it described. The equipment name lived in the folder path, never in the text that was embedded. A paragraph about "the motor" was just a paragraph about a motor.
- 3,344 drawings had silently vanished. Pages with no text layer need a vision model to describe them. That call was failing and the failure was logged as a warning, so ingestion carried on and the page was simply never indexed.
- 260 files were never seen at all. The scanner looked for
*.pdf. They were*.PDF.
None of these is a model problem. Every one of them would have survived a model upgrade.
Building a benchmark from the plant's own manuals
Leaderboards rank embedding models on general text. My corpus is OEM manuals full of torque tables, part numbers and drawings. The only ranking that matters is the one on these documents, so I built one.
Five manuals, every gold page checked by hand, searched against the whole deduplicated corpus so the other 30,000+ chunks act as distractors.
- 31 direct questions, worded close to the page.
- 40 paraphrased questions, worded the way someone on the floor would ask, with the manual's own terms deliberately avoided. A third of them in Spanish, because that is how half the crew asks.
- Recall@8: does the right page make it into the eight sources the answer model sees? With 40 questions the margin is about ±8 points, so anything closer than that is a tie.
voyage-4-large won, Qwen3-Embedding-8B tied it on the hard set at a tenth of the price, and the model I had in production was eight points behind. A real gain, but not the 40-to-78 story.
The header was worth more than the model
The fix for "no chunk knows its machine" is three lines. Every chunk is now embedded behind a header built from its path on the share. For example (names changed):
Equipment: Vertical Roller Mill
Section: Volume 3 / Tab 4
Document: lubrication-system.pdf, page 12
The stored text does not change, so citations and snippets look exactly as before. Only the vector moves, and it moves toward the right machine.
Four answer models, and the question that separates them
Retrieval decides what the model sees. The answer model decides what it does when what it sees is wrong. I gave four models the same eight sources for all 71 questions and had a fifth model grade them blind against the gold page.
When the right page was among the sources, all four scored the same, about 1.8 out of 2. The gap is entirely in the other case: the 19 questions where retrieval failed and the right page was not there.
A wrong motor rating with a citation is worse than no answer, because it looks checked. That is why the answer model changed to Claude Haiku 5.5, at the same cost per answer as before: about a tenth of a cent.
The drawings were never there
The April article said a vision model described every diagram. It did not. The root cause turned out to be one parameter: the vision call had a budget of 1,000 output tokens, and the reasoning model spent all of it thinking. It returned an empty description, the page had no other text, and the page was dropped without an error.
Vision now runs on Claude Haiku 5.5 with thinking off and a 4,000-token budget. On an A0 outline drawing it transcribed the legend, dimensions and loads correctly in 12 seconds for about $0.0013. The old model, even at low effort, took 41 seconds and four times the price.
Before re-reading three thousand drawings I fixed the duplicates, because there is no point paying to describe the same page twice:
- Every page now carries a fingerprint: its normalized text, or a small grayscale render when it has no text. Ingestion skips any page already indexed anywhere, before vision and before embedding.
- Binders go first, then tab files, then the grey copies, so the complete document wins.
- The cleanup removed 11,653 duplicate pages from the existing index without losing any content: 72,748 rows in 452 documents became 45,042 in 282.
The incremental ingest after that found 1,446 pages that needed vision and 539 new text pages across 217 PDFs, ran for 1 hour 26 minutes and finished with no errors. Drawings now show up among the sources of real searches.
Letting the crew add what is missing
The manuals are not the whole story; the knowledge in the plant is also in photos, marked-up prints and hand sketches. So the app now takes uploads:
- PDF, JPG or PNG. A photo becomes a one-page PDF, upright, so the viewer and citations work the same way.
- An equipment field, which becomes the folder and therefore the
Equipment:line in the header. The header pays off twice. - A transcribe-first vision prompt for uploads: write down every word on the page, handwriting included, then describe it.
- A shared upload key for the people I trust with it, on top of the sign-in. Deleting and re-scanning stay admin-only.
The first real upload was a scanned sheet of hand sketches for a counterweight and its guides. It came out as 12 searchable chunks.
Moving it into the cluster
The app also moved out of a Docker Compose VM and into my k3s cluster, and the rule was to reuse what already exists:
- The database went into the CloudNativePG cluster that already serves another app, as its own database and role, instead of a new Postgres. The vector index is 413 MB after a reindex.
- The PDFs stay where they were, on the storage container's NFS export; the pods mount it directly.
- Sign-in is an emailed six-digit PIN for the company's two email domains, the same pattern as my other work apps. No public page, nothing to guess.
- A weekly CronJob dumps the database to the backup share, and the nightly restic job carries it off site.
What it cost
| Item | Cost |
|---|---|
| Model pilot: four embedding models over 41,000 chunks, four answer models, a blind judge | ~$6.55 |
| Voyage tokens for the header experiments | ~$3.15 |
| Re-embedding production with voyage-4-large | ~$2.80 |
| Vision on 1,446 drawings with Claude Haiku 5.5 | ~$2.00 |
| Total | ~$14.50 |
| Each search afterwards, answer included | ~$0.001 |
What I'd actually take from this
- Measure on your own documents. Leaderboards rank models on general text. Mine is torque tables and part lists, and the order changed.
- Fix the data before the model. Duplicates and missing context cost more than any model choice, and a model swap would have hidden both.
- Test the failure case on purpose. Every answer model looks the same when retrieval works. Choose the one that admits it did not find the page.
- Keep the old setup as the control. "Better" means nothing without the number it beat.
- Log failures as failures. Three thousand drawings disappeared behind a warning.
The search box looks exactly the same as it did in April. What changed is how often the page behind the answer is the right one.