Vector-search: how many vector queries per second can one CPU box really do?

Jul 1, 2026(2 months ago)-

Fuxing Loh

A while back (a few weeks from writing this article) I ran into a vector-search problem space, and my mind naturally gravitated to it due to my background in the spatial-search space. The shape of it felt familiar, index and partition the data, route the query to indexes, rerank the candidates after fusion.

But there’s one difference that changes everything. In spatial, the partition is exact: a geohash prefix is the answer to “what’s near here”, so the candidate set is bounded and the query cost is known before you run it. In high-dimensional embedding space there is no prefix. Every cheap way to partition the space is lossy, the true neighbor can always be sitting in a cell you didn’t visit, so correctness stops being a guarantee and becomes a dial called recall, and the known-O(complexity) lookup I was used to becomes a scan O(n) problem that you have to somehow make it fast still.

How everyone else does it

Since nobody does an exact router, everyone approximates, and part of the study was reading and understanding how the market players balance recall / throughput / latency / cost. Once you line them up, the whole landscape is two decisions.

Decision one: skip structure. How do you avoid touching most of the corpus? Either you cluster them into IVF cells and route each query to a few calls, or you build a proximity graph and greedy-walk toward the neighborhood. This choice then becomes where the vectors live: a graph walk is a chain of dependent random hops, so it wants everything in RAM or NVMe (fast random access); While IVF cells are contiguous blocks you can stream, so they can be reasonably done on spinning disk and object storage.

Decision two: scan representation or quantization. Once you’re down to candidates, what do you actually compare? Full-precision floats (exact but fat), or compressed codes with a full-precision rerank to buy the recall back.

Some of the engines I studied, and their choices:

Read the choices column by column and one pattern is clear. Nobody past toy scale scans floats, everyone compresses and reranks, the argument is only over which codes. The graph engines are betting on RAM and latency; the IVF engines are betting on cheap storage and throughput; Weaviate’s flat+BQ mode is the more interesting design choice—that below a certain corpus size, the fanciest structure loses to a brutally fast scan. And whichever skip structure they pick, every one of them ends up needing the same inner layer, scan compressed codes fast (quantized or binary) and, rerank a shortlist exactly (full FP). That layer is what this project is, the scan inside that one IVF cell pushed as far as the hardware allows, with the router above it deliberately out of scope.

The detail I only appreciated later: Weaviate’s flat+BQ and Qdrant’s oversampling+rerank are the exact binary funnel this study converged on independently.

Why I dug in? (Why I’m interested in wasting 2 weekends on this?)

Today every LLM application sits on top of this, retrieval is embeddings, and embeddings mean top-k vector search at production serving latency. So I had two reasons to dig in (waste my weekends). The first is past experiences and curiosity. The second is that this problem is hardware-adjacent to LLM inference itself: a vector scan streams a big block of memory and does a little math per byte, which is the same shape as attention streaming a KV cache, and both slam into the same bottleneck: memory bandwidth, not compute. Studying the small version of the problem, where I can measure everything on one box, felt like the easiest way to understand the limitations the big one is running into. I could have just read the papers and studied how everyone else does it, but applied research (easy with LLM now) is how things actually sticks anyway, build it, vibe it, measure it, try something and be wrong, then understand it.

So I built this project (NNDB), an in-memory vector-search engine from scratch in Rust. Not to ship a product, but as a curiosity run (using /goal and /loop to help me achieve it, run on Opus 4.8, if you’re curious). The thing I really wanted to understand is the boundary between memory-bandwidth-bound and CPU-bound, because that boundary is where all the interesting tradeoffs live. In NNDB, every change became a numbered experiment with the measurements that justified or killed it, and the repo essentially is a lab notebook. 77 experiments later the study is done, and the full writeup with interactive charts is at vector-search.fuxing.dev.

Here’s the TLDR up front: you can always get vector search under 10ms if you really want to. The question is what it costs you. Latency is purchasable with compute dollars, so essentially the real study is the tradeoff of recall against QPS against bytes against silicon, and what the best implementation looks like once you understand which wall you’re actually hitting.

Basically it’s just a unit economics problem, and the three levers are:

1. Recall vs QPS (higher recall lower QPS, lower recall higher QPS)
2. QPS vs bytes streamed (higher QPS lower bytes, lower QPS higher bytes)
3. Bytes streamed vs silicon parallelized (more bytes more silicon, less bytes less silicon)

The bottleneck is just memory, not compute (duh.)

Brute-force f32 search over 1M Cohere embeddings does about 9 QPS. My first instinct was to speed up the math, make the distance computation itself faster with SIMD and FMA, and it worked for a bit. But the bottleneck was never compute. Every query streams 4 GB from RAM, and you can’t out-compute the memory bus. So it’s clear, everything collapsed into two levers: send fewer bytes, or do fewer instructions on each byte.

Where it ended up

Where it ended up is a 1-bit binary funnel. Keep one sign bit per dimension, which is 32× smaller, scan everything with popcount to get a shortlist, then re-rank only the shortlist against the real vectors. Comparing two binary codes is an XOR and then count-the-differing-bits, and counting bits is a single instruction in the CPU’s SIMD set. VPOPCNTDQ chews through 512 bits at a time. That’s why the scan is so damn fast: the entire per-vector comparison is one instruction, no multiplies, no floats. Previously, the same box that did 9 QPS now does ~960 QPS at 0.995 recall, and the whole index is 128 MB. An AMD Zen5 box at the same price (more cores/memory bus) does ~2,310. A full 192-core socket gets to ~19,900 and then the memory bus fully saturates. On 256-bit Matryoshka codes the same funnel does 0.991 recall at ~5,000 QPS with a 3 ms p50.

So what?

Pareto curve/point, in simple english:

• Find the resource you are paying for most, and use all of it. (Memory Bandwidth)
• Then go for the next limit and optimize it along the curve. (CPU SIMD Vectorized math)
• Decide what’s the acceptable trade off.

I tried a scheme with better recall-per-bit and it lost to plain popcount anyway, because its inner loop was a gather the CPU can’t vectorize. That observation (SIMD basically) decided almost every call after it. The other thing I learned is that shrinking the data re-prices every trick you’ve already tuned, because it moves you across the bandwidth-bound/compute-bound boundary. Query tiling bought me +73% while the scan was bandwidth-bound, then cost 24% once the codes fit in cache. Each tuning verdict doesn’t survive when the data structure changed, and it flipped the pareto point.

Pushing the frontier

Once the L3 cache fully fitted (128MB), it also meant the lab had stopped resembling the real problem, so I’ve to push the corpus to 10M vectors, 320 MB of codes, far past any cache, and expected to be back at the DRAM wall. But the problem was actually compute-bound, which means the kernel itself was the only thing left to saturates. Arc 067–074 of the experiments are tail-end optimization, reading the actual disassembly and doing autopsies on the emitted kernel. It turned out roughly half the hot loop’s uops were safety scaffolding, bounds checks, per-lane extractions, re-fetched loop invariants, and the binding resource on Zen5 wasn’t even the ALUs, it was dispatch width, 8 uops per cycle. Once the safe-Rust tricks ran out, the vector units fully saturated doing the math instead of proving to the compiler that the math is allowed.

In summary, kernel-only changes with recall bit-identical at every step: The QPS moved from 650 → 1,471, at 2.26×. The loop is now ~20 uops per document and essentially irreducible at this ISA.

Even at “irreducible” there’s more hyper-optimization on the table, because the two walls take different levers, and you always end up leaning on one of them. When the memory bus is the wall, the lever is bytes streamed: “better” embedding models are effectively a compression scheme, Matryoshka took the codes from 1024 bits to 256 without losing the neighbors, and a model trained to survive 128-bit binary quantization would halve the stream again. When compute is the wall, the lever is tiling: fit as many queries into a tile as the registers and cache allow, so every code loaded is scored against a full tile and the vector units never wait on the next byte. The optimization never end, it just alternates which wall you’re pushing—CPU to DRAM, DRAM back to CPU.

Why it stops here, for me.

What ends the study isn’t running out of frontier or novels hyper optimization, it’s that the next questions aren’t answerable from a benchmark box. The real problem is billions of vectors that will never all be hot at once, and at that scale the problem shifts from making one scan fast to orchestration: noticing which IVF cells are hot, keeping those resident in cache while the cold ones live in RAM or on disk, deciding what to evict and when. I can’t test any of that in a “lab environment”, because the answers depend on a real access pattern, how the corpus grows, and what recall the product actually needs.

Having a 192-core single-box perfected is easy, throughput is predictable from cores and corpus size, the bandwidth wall can be mapped, even having a rule for which instance to buy, over-optimizing a kernel with nothing left to push. But production is chaotic, real usage is not lab-like.