# VoiceMem: the reply model is a QLoRA that can't touch 93% of its own base
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/voicemem
> date: 2026-08-27
> tags: voice-agents, agent-memory, lora, mixture-of-experts, retrieval, speech
VoiceMem showed up under three names before I had read a single equation. The paper on arXiv is titled *[Streaming Dual-Brain Memory for Real-Time Interaction](https://arxiv.org/abs/2608.26005)* — nothing there says QLoRA, MoE, or 35B. The Hugging Face repo I was pointed at is `zhifeixie/VoiceMem_MF_Qwen3_6_35B_A3B_Qlora`. Its own README opens with a note that the repo it is *actually* attached to is misnamed and used to be called something else entirely. Three artifacts, three identities, and — this is the part worth taking seriously — genuinely useful engineering underneath all three, once you separate what each one actually claims.
The short version: VoiceMem is a retrieval architecture, not a model. It routes a streaming transcript through a two-level schema/entity index (the "left brain") and a persona graph (the "right brain"), so a backend search — the paper uses Mem0 — only has to rank a small, semantically bounded candidate pool instead of the whole memory store. That's the actual mechanism, and Section 3's equations back it with real numbers: top-5 retrieval at 430 tokens and 134ms, flat across two orders of magnitude in K. The "reply model" I was also asked to dig into — a rank-32 QLoRA adapter on `Qwen/Qwen3.6-35B-A3B`, a 36B-parameter MoE — is a much smaller and much less-documented piece: it appears exactly once in the paper, as a two-word table footnote, and every architectural detail about it comes from Hugging Face, not the PDF.
| | |
|---|---|
| Paper | [arXiv:2608.26005](https://arxiv.org/abs/2608.26005) — *VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction*, Xie et al. (NTU / NUS / Tsinghua / CUHK), 26 Aug 2026 |
| Mechanism | schema→entity one-hop candidate pool (left brain) + independent/cross-entity persona graph (right brain), both backed by an interchangeable retrieval engine (Mem0) |
| Headline retrieval result | 91.2% on LoCoMo at K=5, 430 tokens, 134ms — vs. Mem0's own default retrieval at 61.68%, 6,956 tokens |
| The "reply model" | `zhifeixie/VoiceMem_MF_Qwen3_6_35B_A3B_Qlora` — rank-32 QLoRA, `Qwen/Qwen3.6-35B-A3B` (36.0B params, MoE, 30 linear-attn + 10 full-attn blocks) |
| Trainable surface | 44,954,880 params (verified from the adapter's own safetensors header) — 0.130% of the base tower, 0% of its 256 routed experts |
| Training corpus | `zhifeixie/VoiceMem-ChatMem400k` — the Hub repo contains a license file and nothing else |
| Checked and holding | "beats Mem0 by nearly 30 points" — confirmed, arithmetic and all |
| Checked and not holding | the abstract's own headline persona number — arXiv's indexed copy says 4.29, the PDF says 1.89 |
## What "dual-brain" actually means
Strip the neuroscience framing and Section 3 is a fairly ordinary indexing problem solved carefully. The left brain organizes memory into a two-level graph — schemas for coarse routing, entities for the concrete people/events/concepts underneath them:
$$
\mathcal{G}^L = (\mathcal{S}, \mathcal{V}, \mathcal{E}), \qquad v = (d_v, \mathcal{N}_v^{\text{micro}}, \mathcal{I}_v), \qquad s = (d_s, \mathcal{N}_s^{\text{macro}}, \mathcal{V}_s)
$$
Every entity belongs to exactly one schema; edges are one-hop and split into "strong" and "weak" so expansion stays local. Retrieval never touches the full memory store `M`. It matches the partial transcript against schemas and entities, expands one hop, and only then searches:
$$
(\mathcal{V}_t, \mathcal{S}_t) = \text{Match}(x_{\le t}, \mathcal{V}, \mathcal{S}), \qquad \mathcal{Z}_t = \mathcal{V}_t \cup \mathcal{V}_{\mathcal{S}_t} \cup \mathcal{N}_1^{\text{strong}}(\cdots) \cup \mathcal{N}_1^{\text{weak}}(\cdots)
$$
$$
\mathcal{C}_t^L = \bigcup_{z \in \mathcal{Z}_t} \mathcal{I}_z, \qquad \mathcal{R}_t^L = \text{MemSearch}(q_t, \mathcal{C}_t^L; K)
$$
That's the whole trick, and it's the same trick a Lucene practitioner would call "query expansion into a filtered index" — the paper's contribution is making it streaming and pairing it with a persistence rule so the index doesn't degrade. As a cluster of entities grows, the paper measures query coherence over a sliding window of sessions —
$$
\rho(H) = \frac{1}{|Q|}\sum_{q \in Q} \frac{|A_q \cap H|}{|A_q \cup H|}
$$
— and an LLM judge promotes a coherent, over-threshold subcluster into its own schema (Algorithm 1 in the paper). That's the mechanism that keeps a schema from turning into a junk drawer as a user's history grows into the thousands of turns: it splits before it degrades, rather than after.
The right brain runs in parallel over a persona graph with two node types — independent nodes for durable traits, cross-entity nodes tied to something in the left brain (so "he feels lonely" and "he feels lonely about Bob" are different, retrievable facts):
$$
\mathcal{G}^R = (\mathcal{V}^I, \mathcal{V}^C), \qquad v^I = (d^I_v, \mathcal{I}^I_v), \qquad v^C_e = (d^C_{v,e}, \mathcal{I}^C_{v,e}, \rho_{v,e}),\ e \in \mathcal{V}
$$
Short-horizon attribution runs per-turn (`et = φ(xt)`, then a graph edit); long-horizon attribution runs per-session, consolidating a sequence of `(x, e)` pairs into stable independent nodes rather than accumulating every transient emotional blip. The full pipeline — preprocessing, both graphs, four interchangeable backend stages — is one diagram in the paper, and it's worth looking at directly rather than through my redrawing of it:
## Why K=5 doesn't need K=30
The number the abstract leads with — top-5 beating classical systems at their own larger budgets by "nearly 30 points" — is really a statement about what the candidate pool does to the ranking problem. Section 5.4 gives four real (K, tokens, accuracy) triples on LoCoMo, and they tell a cleaner story stacked together than any one of them does alone:
The ablation is the useful row here. Turn schema routing off and keep everything else — same backend, same ranker — and VoiceMem still reaches the same accuracy the routed version gets at K=5. It just needs K=30 and three times the tokens to search its way there, because now the ranker is working over the whole one-hop expansion rather than a schema-bounded subset of it. That is a clean, checkable version of "routing doesn't raise the ceiling, it lowers the budget needed to reach it," which is the paper's own phrase for it, and it's a more honest framing of the "beats Mem0" claim than the abstract's version — the ceiling most of these systems can eventually reach with enough budget is closer together than the top-line numbers suggest; what actually separates them is how much you have to spend to get there.
Bounding the candidate pool is also what makes the latency number hold across K, which matters more than the accuracy number for a system whose whole premise is fitting inside a live voice loop:
The three streaming stages (Eq. 8–10 in the paper) overlap listening rather than following it, so only the last ~100ms of the nominal 500ms VAD window is dedicated search time — and the paper measures that search at 134ms, comfortably inside the window even though it slightly overruns its own diagram's stage boundary. What's genuinely nice engineering is *why* it stays flat: schema routing fixes the size of the pool before ranking runs, so widening K reranks the same small set rather than searching a bigger one. A flat-context system without that bound would see search time grow with K, and most of the baselines in the paper's own comparison do.
One retrieval claim in the abstract — "outperforms classical systems such as Mem0 at top-200 by nearly 30 points" — is one I could check directly and it holds up. Table 4 gives Mem0's own default retrieval at 61.68% versus VoiceMem's schema-routed index over the same Mem0 backend at 91.20%: a 29.52-point gap, "nearly 30" exactly. Figure 6's K-sweep confirms it isn't a cherry-picked operating point — Mem0's own curve never clears 63% at any K from 1 to 200. Not every headline number in this paper survives contact with the source (more below), but this one does, cleanly.
## The reply model: a QLoRA the paper barely mentions
Section 4.1 describes training three model families into memory-grounded speech models through "online black-box on-policy distillation": Qwen2.5-Omni, Qwen3-Omni, and Step-Audio2-Mini, each distilled from a larger sibling. `Qwen3.6-35B-A3B` — the base of the adapter this task actually pointed me at — is not one of them. It shows up in the paper exactly once, as a footnote on Tables 1 and 2: *"‡: responses generated by our fine-tuned Qwen3.6."* No rank, no target modules, no training data description, no mention of LoRA or QLoRA anywhere in eighteen pages. Everything below comes from Hugging Face, not the PDF — I'm keeping the two sourced separately on purpose.
`Qwen/Qwen3.6-35B-A3B` is a real, popular Qwen release (5.2M downloads on the Hub) and the identical architecture this site has already profiled once, for [Ornith-1.5-35B-A3B](/articles/ornith-1-5) in the [Tiel-Coder](/articles/tiel-coder-35b-a3b) piece: 40 blocks, 30 linear-attention (gated-delta-style) and 10 full-attention, 256 experts routed 8-per-token, 2,048 hidden. I re-derived the parameter census independently, straight from the safetensors headers via HTTP range requests (the header is a length-prefixed JSON blob sitting at byte 0, so you never download the weights to read it) — and it reproduces the earlier article's numbers to the last digit: 34,660,610,688 in the language-model tower, 2,946,429,568 active per token. Whatever else differs between these two releases, the base tensor layout is the same file, twice.
`adapter_config.json` ships a QLoRA with `r: 32`, `lora_alpha: 64`, `lora_dropout: 0.05`, and this `target_modules` regex:
```
^(model\.language_model(?=\.).*\.(shared_expert_gate|down_proj|out_proj|
in_proj_a|in_proj_b|q_proj|in_proj_z|gate_proj|up_proj|in_proj_qkv|
k_proj|v_proj|o_proj))$
```
Read that regex and you'd expect it to touch the mixture: `down_proj`, `up_proj`, and `gate_proj` are exactly the names on the 256 routed experts that make up 92.9% of the tower. It doesn't. The base model stores every block's experts as one fused tensor — `mlp.experts.down_proj`, shape `[256, 2048, 512]` — not 256 separate `nn.Linear` submodules, and PEFT's default LoRA attaches to module boundaries, not tensor names. Reaching a fused expert tensor needs PEFT's `target_parameters` field, which this adapter's config sets to `null`. So the regex's expert-shaped words describe the shared expert (one small dense MLP per block, always active) and nothing in the sparse mixture.
I didn't take PEFT's word for what the regex resolves to — I read the adapter's own `adapter_model.safetensors` header the same way I read the base model's, and it lists exactly 700 tensors (350 targeted modules × `lora_A`/`lora_B`) summing to 44,954,880 parameters, all F32. That's 350 modules across 40 blocks — the four/five attention or linear-attention projections per block, plus the shared expert's three matrices and its gate — and it reproduces the file size on its own: 44,954,880 × 4 bytes = 179.8 MB against the repo's stated "180 MB" and the file's actual 179,929,048 bytes. Every number in the widget above is checkable the same way; none of it is an estimate.
## The corpus: 400K of something, in a repo with nothing in it
`CHATMEM-400K` is described in Section 4.1 as the output of a four-stage loop — synthetic persona/background/event construction, then an iterative "Task & Topic → Scene → Challenge & Strategy → Contrastive Distillation → Verification → SFT Update" cycle that the paper calls SLM-verified online distillation. That's a real, specific pipeline description. What it never states, anywhere in eighteen pages, is what unit "400K" counts — conversations, turns, or samples. The one dataset the paper *does* give exact stats for is the much smaller human-curated eval split, ChatMem-Bench: 316 questions drawn from 15,314 turns and 53 hours of audio, across 14 categories in four dimensions (Information, Persona, Affective Attribution, Paralinguistics & Environment).
I went to `zhifeixie/VoiceMem-ChatMem400k` on Hugging Face to resolve it from the data itself. The repo's file listing is `.gitattributes`, `README.md` — nothing else. The README is three lines: a YAML front-matter block declaring an Apache-2.0 license, and no description, no dataset card, no schema, no sample rows. There is no dataset here to inspect. The QLoRA model card is upfront about the same gap on the training side — *"Training data are not released. Provenance, licenses, and consent status are documented in the code repository before any data release"* — and the linked code repository (`lang-jiaqi/Voicemem_open`) wasn't reachable from this environment to check whether that documentation exists yet. So: I can tell you precisely how the corpus was supposed to be built, and I can tell you precisely that nothing behind either public name — the dataset repo or the code repo's promised docs — currently lets you verify what "400K" means.
## Checking the abstract against itself
This is the discrepancy I didn't expect to find. arXiv serves two versions of this paper's abstract for the same submission: the HTML `/abs` page, its `og:description`, and its `citation_abstract` metadata all read *"improves the aggregate score by 4.29 points over the previous best system."* The actual PDF — rendered to an image and read directly off the page, not OCR'd — prints a different number in the identical sentence: *"improves the aggregate score by 1.89 points over the previous best system."*
Table 2 explains exactly where both numbers come from. MemOS, the strongest baseline, averages 72.27 across eleven persona sub-categories. VoiceMem replying through GPT-4o-mini — the same generator every baseline in the table uses — averages 74.16, a gap of 1.89. VoiceMem replying through its own fine-tuned model instead averages 76.56, a gap of 4.29. Both deltas are real rows in the same table; neither is a typo. But they answer different questions. 1.89 is the matched-generator number: hold the responder fixed and change only the memory system, which is what "the right brain improves persona memory by N points" ought to mean, and it's the number the printed PDF states. 4.29 additionally swaps in a better generator for VoiceMem alone — a real, footnoted comparison, but not evidence about the memory architecture on its own. It's also, of the two, the number arXiv's own indexed abstract shows to Google Scholar, Semantic Scholar, and every social-card preview. Anyone who cites the abstract without opening the PDF is citing the less conservative of the two — through no fault of their own, since arXiv is where the "quotable" text lives.
Two smaller things worth naming plainly rather than letting slide:
The QLoRA model card's own opening note reads: *"The repository name does not match its contents. This repo is named `VoiceMem_SLM_Qwen25_omni`, but the weights published here are the Qwen3.6-35B-A3B reply adapter."* The Quickstart code block a few lines down still sets `adapter_id = "zhifeixie/VoiceMem_SLM_Qwen25_omni"` — a repo name that isn't the one this page lives at (`VoiceMem_MF_Qwen3_6_35B_A3B_Qlora`) and, as far as I could find, isn't a real repo either. Copy the Quickstart verbatim and it points at nothing.
The card's own standalone benchmark for this adapter — "AudioMC `INFERENCE_MEMORY`," 132 conversations, 233 rubric criteria, judged by GPT-4o-mini — doesn't appear anywhere in the paper; it's a model-card-only number I can't cross-check against the arXiv source at all. To its credit, the card doesn't oversell it: `checkpoint-3318` satisfies 97 of 233 criteria against GPT-4o-mini's 96, a +0.43 percentage-point edge the card itself calls "a small margin... not evidence of a large capability gain." That's an honest way to report a result this close to noise, and it's the right instinct — I just can't verify the benchmark itself, only that the card doesn't inflate what it found.
Also worth being precise about, since it's easy to assume otherwise from the "voice agent" framing: `Qwen3.6-35B-A3B`'s own `config.json` declares `image_token_id` and `video_token_id` but no audio modality at all. This specific adapter is a text-in, text-out reply model that consumes an already-retrieved memory context (per the model card: "the context is Top-K retrieved memory, not raw conversation transcripts") — the audio path (ASR, voiceprint, emotion recognition) is the separate streaming preprocessing Section 3.3 describes, running upstream and outside this repo entirely.
## The ledger
**Genuinely good work.** The candidate-pool bound in Section 3.1 is a real, checkable idea, not a rebrand of top-K retrieval — the one-hop schema/entity expansion is what keeps K=5 competitive with K=30-100 on other systems, and the paper's own ablation (routing on vs. off, same backend) isolates exactly what it buys: 3x fewer tokens for the same accuracy, not a higher ceiling. The streaming split into speech-tail/anticipation/searching is a sensible way to hide preprocessing latency inside a window the user experiences as silence, and the 134ms search figure surviving flat from K=3 to K=100 is the direct, load-bearing consequence of bounding the pool before ranking. The MMLU-Pro-style discipline of publishing the routing-disabled ablation, rather than only the final number, is the kind of thing worth crediting.
**What is convergent, not novel.** Schema-then-entity indexing over a vector backend is a fairly standard two-level RAG pattern; the persona graph's independent/cross-entity split is a sensible but not unprecedented way to separate traits from situational affect. None of that is a knock — the paper doesn't claim otherwise, and getting a standard pattern to run inside a 500ms voice-turn budget is real systems work.
**What I would watch.** Whether `ChatMem-400K` and its promised provenance documentation actually land in the code repository, since right now "400K" is unverifiable by anyone outside the author group. Whether the QLoRA reply model — genuinely absent from the paper's method section — gets folded into a future revision with the training details the model card is missing. And whether arXiv reconciles its own indexed abstract with the PDF it's serving; a 2.3x discrepancy on the paper's own headline persona number, sitting in the metadata every citation tool reads, is the kind of thing that should get fixed by an errata rather than discovered by someone reading pixels off page 1.
For the retrieval-architecture side of this, [TencentDB's agent-memory system](/articles/tencentdb-agent-memory) and [MemHarness](/articles/memharness) are the closer comparisons on this site — both are also memory systems judged on what happens when you check the plumbing rather than the headline table. For the parameter-arithmetic side, [Tiel-Coder-35B-A3B](/articles/tiel-coder-35b-a3b) and [Whittle MoE](/articles/qwen3-8-whittle-moe) go through the same "read the tensor shapes, not the config's claims" exercise on sibling Qwen MoE releases, and if the sparse/dense split above was new to you, [Mixture of Experts, from scratch](/articles/mixture-of-experts-from-scratch) builds up why routing works that way at all.