TurboVec: FAISS-competitive vector search with no training phase

mdjsonmcpcopy↗ai

2026-07-02 · 6 min · vector-search · retrieval · quantization · rust

Why read this

Niche· last 15%

TurboVec explained as TurboQuant applied to vector search: no train() step, the six-step encode, its SIMD scan, and where it beats or trails FAISS IndexPQ.

• Runs on a laptop CPU
• Concrete numbers to act on

·Developer tools & infra·MIT·Practitioner tool

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

0 of 3: Restates claims

Can I run it?

3 of 3: Open, permissive, runs on reader hardware with instructions

Will I understand it?

1 of 3: Partial mechanism

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

0 of 3: Same as the press release

Score 37 of 100, ranked 548 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Vector search is memory-bound. A few million embeddings at float32 are gigabytes of RAM, and the standard fix — product quantization (PQ), the workhorse inside FAISS — compresses them by learning a codebook: run k-means over a sample of your data, replace each sub-vector with the nearest centroid's index. It works well, but it has a cost that's easy to forget: a training phase. You need representative data up front, you call train() before you can add anything, and as your corpus drifts the learned codebook goes stale and wants a rebuild.

TurboVec (RyanCodrai/turbovec) removes the training phase entirely. It's a Rust vector index built on TurboQuant — the same rotate-then-quantize algorithm that compresses LLM KV caches — whose quantizer is data-free, so there's nothing to learn. The headline: a 10-million-vector corpus that costs ~31 GB as float32 fits in ~4 GB, and searches faster than FAISS.

Why there's nothing to train

The full derivation is in the TurboQuant write-up, but the short version is the whole reason there's no training step. TurboVec encodes each vector in six moves:

1. Normalize to the unit sphere (the length is stored separately for scoring).
2. Rotate by a random orthogonal matrix — built once from a seeded Gaussian via QR, so it's deterministic and, crucially, data-independent.
3. TQ+ calibration — a small per-coordinate shift/scale, fit once on the first batch, to snap real embeddings onto the ideal post-rotation marginal.
4. Lloyd–Max scalar quantization — the optimal quantization levels, precomputed against the known Beta distribution the rotation produces. This is the data-free part: because you know the distribution analytically, the optimal quantizer is a fixed table, not something learned from your vectors.
5. Bit-pack to 2, 3, or 4 bits per coordinate — 16× smaller than float32 at 2-bit.
6. Store a per-vector correction scalar so the quantized dot product is an unbiased estimate of the true inner product (a RaBitQ-style length renormalization).

PQ learns its codebook from your data; TurboQuant reshapes your data into a distribution whose optimal codebook is already known. That's the trade — and it's why TurboVec can ingest online with no train(), no parameter tuning, and no rebuilds as the corpus grows.

expand

Does it actually beat FAISS?

Mostly yes, and the repo is honest about where it doesn't. Across its benchmark configs (100K vectors, 1K queries, k=64), here's TurboVec against FAISS IndexPQ on recall, latency, and compression — flip through the datasets and bit-widths:

On the OpenAI embedding sets it wins recall@1 outright (up to +1.9 points at 2-bit) and runs 12–19% faster on ARM, at 8–16× compression with no training pass. The exceptions are real: on x86 it trails a few percent at 2-bit (it wins the 4-bit configs), and on low-dimensional GloVe vectors at 2-bit FAISS edges it by 0.06 of a point — the rotation has less room to spread energy in only 200 dimensions. Net, it's genuinely competitive with a mature, heavily-optimized library, which is a high bar for a quantizer with no learned codebook.

expand

The systems half

A near-optimal quantizer only matters if the search is fast, and TurboVec is a real systems project, not a reference implementation:

• Hand-written SIMD kernels — NEON on ARM, AVX-512BW on x86, with an AVX2 fallback and runtime feature detection. Scoring runs on the packed codes directly via table lookups; there's no decompression step.
• 32-vector blocks, FAISS FastScan-style, with the query LUT built per search. Filtering is a bitmask checked at block granularity — whole blocks with no allowed vectors are skipped, and the filter is applied inside the kernel so a restricted search returns the true top-k among allowed items with no recall penalty and no over-fetch.
• IdMapIndex gives stable uint64 external IDs with O(1) removal (swap-remove, no tombstones) — the vector you delete is replaced by the last one and both ID maps update in constant time.
• Online ingest and plain persistence (.tv / .tvim files), plus drop-in adapters for LangChain, LlamaIndex, Haystack, and Agno.

The Python API is what you'd hope for — no training call anywhere:

from turbovec import TurboQuantIndex
 
index = TurboQuantIndex(dim=1536, bit_width=4)   # 2, 3, or 4 bits
index.add(vectors)                                # float32 (n, dim) — indexed immediately
scores, ids = index.search(query, k=10)           # searches the packed codes
index.write("corpus.tv")

Where it fits (and where it doesn't)

The honest scope: TurboVec is a flat, exhaustive-scan index — it scores every (packed) vector per query. That's exactly the regime where it competes with FAISS's flat PQ scan, and at a hundred thousand to a few million vectors it's excellent: no training, tiny memory, strong recall. It is not a billion-scale graph index — if you need sub-linear search over hundreds of millions of vectors you still want an IVF or HNSW structure (and you could quantize those with TurboQuant too). Think of it as the compression-and-scoring core done unusually well, not a replacement for every ANN system.

The take

What I like about TurboVec is that it makes the TurboQuant thesis concrete in a second domain: the same "rotate into a known distribution, then quantize with a data-free optimal quantizer" that shrinks KV caches also shrinks embedding indexes — and here it buys something PQ structurally can't, the elimination of the training phase. Pair it with a lexical scorer like BM25 and you've got both halves of hybrid retrieval, each running on commodity hardware with no GPU and no learned index. It won't dethrone HNSW at billion scale, but for the very common case of a few million embeddings that need to fit in RAM and update live, "as good as FAISS, with nothing to train" is a genuinely nice place to land.

Built on RyanCodrai/turbovec (Rust + Python, MIT), which implements TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (Zandieh, Daliri, Hadian, Mirrokni; ICLR 2026). Benchmark figures are from the repo's published results (100K vectors, k=64; ARM = Apple M3 Max, x86 = Xeon Sapphire Rapids).