Search Can't Count: Vector Databases vs. an OKF Index for Finding Every Qualified Supplier
Imagine you run a sourcing platform for manufacturing. Buyers post job specs — a bracket that needs machining, an enclosure that needs molding — each listing eight to twelve requirements: "must machine titanium," "must be AS9100 certified," "must hold ±0.0005-inch tolerances," "must offer in-house anodizing." Machine shops and fabricators upload capability statements describing what they do. Your product's whole job is the match: for each spec, find every shop that can actually make the part, and rank the best fits first.
Miss a qualified shop and they lose a job they never knew existed — and the buyer overpays a shop you did surface. Match an unqualified one and the buyer burns two weeks of quoting before discovering the shop can't hold the tolerance. The catalog grows by hundreds of capability statements a month, and nobody re-reads the old ones.
We built this two different ways and benchmarked them head to head on a corpus where we knew, with certainty, what the right answers were. One approach was the standard playbook everyone reaches for. The other made an LLM read every capability statement once and file what it found. The gap between them was not small, and the reasons why generalize far beyond manufacturing — to any problem shaped like "find everyone in this pile who qualifies."
The standard playbook, and why we trusted it
The default architecture for "search a pile of documents with AI" looks like this: store the documents, cut each one into chunks, and turn each chunk into an embedding — a list of numbers that acts like a fingerprint of the chunk's meaning. Text with similar meaning gets similar fingerprints, so a search for "titanium machining" can find a paragraph that says "we cut exotic alloys for flight hardware" even though they share no words. That's the magic plain keyword search can't do, and it's why this pattern — usually Postgres with the pgvector extension, plus classic keyword search as a backstop — is the industry default.
We ran exactly that: every capability statement split into sections, every section embedded with a top-shelf embedding model, both a semantic search and a keyword search per requirement, results merged. For each spec, the system searched once per requirement, noted which shops kept showing up across requirements, and ranked them by how many requirements they appeared to cover.
To grade it, we generated a synthetic corpus — 2,000 supplier capability statements, 50 job specs — where every shop's true capabilities were planted by the generator. That means no arguments about judgment calls: for every spec we know exactly which shops qualify. We also planted the traps that real documents contain: shops describing the same capability in different words, capabilities mentioned only inside a customer-story anecdote ("for an aerospace client, we delivered…"), lookalike capabilities that don't count (ISO 9001 is not AS9100, and powder coating is not anodizing), and — the sneaky one — sentences like "we do not currently machine titanium."
What went wrong: search returns the best few, but the answer is a crowd
Here's the number that explains almost everything that follows. In our 2,000-shop corpus, a typical requirement — say, in-house anodizing — is genuinely offered by about 350 shops. A search, semantic or keyword, doesn't return everyone who matches; it returns the top results — we used the top 30 chunks, a typical setting. Thirty chunks came out to roughly 55 distinct shops.
Stop and sit with that: the answer set is 350 shops, and the search window shows 55. It doesn't matter how brilliant the embeddings are. Around 300 qualified shops are invisible on that requirement — not ranked low, invisible — before any ranking logic runs. And because a shop must show up across most of a spec's ten requirements to rank well, the odds compound against it on every search.
This is why the results collapsed as the catalog grew. At 500 suppliers, our best configuration put about a third of the truly qualified shops into the top-10 results. At 2,000 suppliers — same code, same models — it fell to under one in ten. Nothing broke. The catalog simply outgrew the search window, exactly the way a growing city outgrows a phone book that only prints the first page of plumbers.
The obvious fix is to widen the window: look at the top 300 chunks instead of 30. We tried. Coverage improved — and trustworthiness fell off a cliff, because the wider window also swept in every trap we'd planted:
- Shops that explicitly said they don't offer a capability got retrieved for it 64% of the time. To an embedding, "we do not machine titanium" and "we machine titanium" are nearly identical fingerprints. Search reads vibes, not logic.
- Shops offering a lookalike capability — powder coating when the spec demands anodizing, ISO 9001 when it demands AS9100 — flooded in as false matches. For a flight-critical part, that's not a ranking nuisance; that's the wrong certification on a part that ends up on an aircraft.
Narrow window: you miss most of the crowd. Wide window: you can't trust who's in it. There is no setting in between that fixes both, because the problem isn't tuning. Search is built to find the best few examples of something. It is structurally incapable of enumerating everyone who qualifies. Those are different questions that happen to share a text box.
The alternative: have the AI read everything once, then never search again
The second system inverts the whole design. When a capability statement is uploaded, an LLM reads it — actually reads it, the way a junior sourcing engineer would — and files what it finds into a shared index: one page per capability, in the style of a wiki. This is the pattern Google recently standardized as the Open Knowledge Format (OKF) — plain, readable pages that AI maintains and both AI and humans can open and check. The "titanium machining" page ends up holding the complete list of every shop that claims it, each with the quoted sentence as evidence. And because the model reads rather than matches, when it hits "we do not currently machine titanium," it files that shop under excluded — the sentence that fooled the search stack 64% of the time gets handled correctly essentially every time.
Answering a spec now involves no search at all. Look up the page for each requirement, pull the complete shop list, and count coverage. The lists are complete, so nobody is invisible; the growing catalog just makes the pages longer, not the answers worse. And when a buyer phrases a requirement in their own words — "we need someone who can hold half-thou tolerances on flight hardware" — a quick LLM call reads the index's table of contents (72 capability pages) and picks the right page. Choosing one topic from 72 named topics is a task language models are extremely good at; it's a much smaller problem than matching a phrase against 55,000 chunk fingerprints.
The same 2,000-supplier benchmark, same trap-laden corpus, same grading:
| Standard stack (best tuning) | OKF index | |
|---|---|---|
| Qualified shops found (shortlist of 25) | 32% | 99% |
| Shortlist size needed to reach ~90% coverage | 200+ docs, still short | 25 docs |
| "We do NOT offer this" mistakes | 64% of opportunities | ~0% |
| Quality change when catalog grew 4× | fell by ~⅔ | unchanged |
| Time per match query | ~2 seconds | under a millisecond |
That shortlist row matters more than it looks. In a real system you don't trust any index blindly — you hand the shortlist to an LLM for final judgment, and that judgment is priced per document it reads. One approach needs 25 documents to show the judge 99% of the right shops. The other needs more than 200 to show 75%. That's an 8× difference in your per-match LLM bill, and it still silently drops a quarter of the qualified shops — the kind that becomes a support ticket titled "why didn't we show up for this job we're clearly equipped for?"
Nothing is free: what the index costs, and where it's fragile
The reading pass is where all the cost moved. Having a frontier model read 2,000 capability statements cost us about $30 and — this surprised us — the real constraint wasn't money but rate limits: the provider caps how many tokens per minute you can push, so bulk ingestion becomes a queue you wait in. Embeddings, by contrast, cost pennies for the entire corpus. If your documents change constantly, that per-document reading cost recurs, and the math tightens.
The bigger caveat is quality. Our corpus was synthetic and the claims were cleanly written, so the reading pass extracted capabilities essentially perfectly — which means our 99% is the ceiling, not a promise. On real documents the index is exactly as good as the reading, and a claim the reader misses is frozen out of the index until something re-reads the document. The standard stack degrades noisily all over the place; the index degrades quietly at ingest. You'd want spot-audits of the extraction the same way you'd audit a human analyst.
One more honest finding, because it cuts the other way: some of the vector stack's failure was self-inflicted by chunking — the choice of how to slice documents before embedding. Our sections blended ~10 different capabilities into one fingerprint, like describing an entire buffet with a single flavor. Re-chunking one sentence per embedding tripled the index size but dramatically improved semantic matching (on the hardest test, from 28% to 75% of qualified shops found in a 200-doc shortlist). If you're committed to vector search, chunk granularity is the highest-leverage dial you have. It still didn't close the gap — the best-tuned, best-chunked configuration needed a shortlist 8× larger to find fewer of the right shops — because no amount of chunking changes what a top-30 window can see of a 350-shop answer.
The transferable lesson
Ask one question of any AI retrieval feature you're building: is the answer a few good examples, or is it everyone who qualifies?
If it's a few good examples — "find me a relevant clause," "answer this question from our docs" — vector search is the right tool and this article changes nothing for you.
If it's everyone who qualifies — matching, screening, compliance, eligibility — then a search index is the wrong data structure no matter how good the embeddings get, because search samples and your problem needs a census. The fix isn't a better query; it's moving the intelligence to the other end of the pipeline: pay the model to read and file each document once at ingest, so answering becomes looking things up in an index you can open, inspect, and correct — instead of re-interpreting the whole pile through a keyhole on every query.
We keep relearning this shape of lesson in production AI work: the most reliable place for a model is rarely the moment of the user's question. It's earlier, where its judgment can be captured once, checked, and reused forever.
The full benchmark — corpus generator, both pipelines, and every number in this post — runs locally and is reproducible end to end. If you're weighing this tradeoff for a matching or screening product, that's exactly the kind of problem we help teams with.