Search and Training Material From 55,001 Pages of Technical PDFs

The company sells and supports equipment from several manufacturers, and what it knows about those products sits in the manufacturers' documents: manuals, brochures and other technical documents, spread across a support portal and a product hub. Two sources, 2,455 files. Three of those files appeared in both places, so after matching every file by its fingerprint the library came to 2,452 distinct PDFs and 55,001 pages.

Nobody reads 55,001 pages to answer one question. The jobs this library had to serve, answering a buyer's fit question, finding a photo for a post and writing training material, each need one passage or one picture and the page it came from. The pictures were out of reach: an earlier pass over the support-portal documents found the outlines of 38,233 pictures and saved none of them.

I treated the job as a pipeline. Each PDF goes through Docling on rented GPU workers, which returns the text, the tables and each picture it locates with its page, its position and the words around it. Large manuals are read in windows of at most 25 pages, with their page numbers kept. The picture run covered 2,236 documents and 51,972 pages in about nine hours on five workers, for $25.20 of GPU time, about a twentieth of a cent a page. It located 71,030 picture placements and stored each one with its document, page and position. Prose is cut at sentence boundaries toward passages of 50 to 150 tokens. Short neighbors merge only when they sit under the same heading, a single sentence longer than the band stays whole, and tables stay whole. The short band is a design choice: a search hit should point at one spec, not a page of mixed topics. Passages and images are embedded with Gemini Embedding 2 at 3,072 dimensions, and the image vectors come from the pixels themselves, not a caption. Search runs keyword and meaning side by side and fuses the two rankings, so an exact part number and a loosely worded question can both match. I built a RAPTOR layer for the passages. It groups related passages, writes a summary for each group, embeds the summary, and repeats, up to 10 levels. That gives search two directions. Horizontal search ranks everything at one level, passages with passages or summaries with summaries. Vertical search starts at a broad summary and walks down to the passages under it. For the image library, a picture is kept when its short side is at least 400 pixels, the larger copy wins when a picture repeats, and each kept picture is tagged by machine type, job type, scene, season, snow, people and whether it would work as a still, a diagram or video B-roll. 30,490 images cleared that bar and were tagged and embedded. The library can be searched by words, by tag, or by handing it a picture and asking for ones that look like it.

• Docling on RunPod GPU workers
• Sentence-boundary chunking toward 50 to 150 tokens
• Gemini Embedding 2, 3,072 dimensions, text and images
• Hybrid keyword and vector search with rank fusion
• RAPTOR summary tree for horizontal and vertical search
• Convex passage store with page numbers

2,452 PDFs and 55,001 pages counted, each file once. 71,030 picture placements stored with their document, page and position, and 30,490 images tagged, embedded and searchable by words, tag or likeness. The approved guides and product pages written from the library carry source maps that tie claims to the manufacturer PDFs and other sources behind them, so a reader, or the client's own team, can check those claims against their sources.

The company's teaching material and product pages now start from its own source documents instead of a blank page. A writer who needs a photo for a post or a diagram for a guide searches 30,490 tagged images from the manufacturers' own documents, by words, by tag or by likeness.

Save the pictures on the first pass. The earlier pass over the support-portal documents found 38,233 picture outlines and kept none of them, so this run stored each picture it located, 71,030 placements, with its page, its position and the text around it.