{"id":837,"date":"2026-05-03T15:47:42","date_gmt":"2026-05-03T07:47:42","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=837"},"modified":"2026-05-03T15:47:42","modified_gmt":"2026-05-03T07:47:42","slug":"sakana-ai-introduces-kame-a-tandem-speech-to-speech-architecture-that-injects-llm-knowledge-in-real-time","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=837","title":{"rendered":"Sakana AI Introduces KAME: A Tandem Speech-to-Speech Architecture That Injects LLM Knowledge in Real Time"},"content":{"rendered":"

The fundamental tension in conversational AI has always been a binary choice: respond fast or respond smart. Real-time speech-to-speech (S2S) models \u2014 the kind that power natural-feeling voice assistants \u2014 start talking almost instantly, but their answers tend to be shallow. Cascaded systems that route speech through a large language model (LLM) are far more knowledgeable, but the pipeline delay is long enough to make conversation feel stilted and robotic. Researchers at Sakana AI,<\/strong> the Tokyo-based AI lab introduces KAME<\/strong> (Knowledge-Access Model Extension), a hybrid architecture that keeps the near-zero response latency of a direct S2S system while injecting the richer knowledge of a back-end LLM in real time.<\/p>\n

The Problem: Two Paradigms, Two Tradeoffs<\/strong><\/h3>\n

To understand why KAME is important, it helps to understand the two dominant designs it bridges.<\/p>\n

A direct S2S model like Moshi (developed by KyutAI) is a monolithic transformer that takes in audio tokens and produces audio tokens in a continuous loop. Because it doesn\u2019t need to synchronize with external systems, its response latency is exceptionally low \u2014 for many queries, the model starts speaking before the user even finishes their question. But because acoustic signals are far information-denser than text, the model has to spend significant capacity modeling paralinguistic features like tone, emotion, and rhythm. That leaves less room for factual knowledge and deep reasoning.<\/p>\n

A cascaded system, by contrast, routes the user\u2019s speech through an Automatic Speech Recognition (ASR) model, feeds the resulting text into a powerful LLM, and then converts the LLM\u2019s response back into speech via a Text-to-Speech (TTS) engine. The knowledge quality is excellent \u2014 you can plug in any frontier LLM \u2014 but the system must wait for the user to finish speaking before ASR and LLM processing can even begin. The result is a median latency of around 2.1 seconds, which is long enough to noticeably interrupt natural conversational flow.<\/p>\n

\n
\"\"
https:\/\/pub.sakana.ai\/kame\/<\/figcaption><\/figure>\n<\/div>\n

KAME\u2019s Architecture: Speaking While Thinking<\/strong><\/h3>\n

KAME operates as a tandem system with two asynchronous components running in parallel.<\/p>\n

The front-end S2S module<\/strong> is based on the Moshi architecture and processes audio in real time at the cycle of discrete audio tokens (approximately every 80 milliseconds). It begins generating a spoken response immediately. Internally, Moshi\u2019s original three-stream design \u2014 input audio, inner monologue (text), and output audio \u2014 is extended in KAME with a fourth stream: the oracle stream<\/strong>. This is the key innovation point.<\/p>\n

The back-end LLM module<\/strong> consists of a streaming speech-to-text (STT) component paired with a full-scale LLM. As the user speaks, the STT component continuously builds a partial transcript and periodically sends it to the back-end LLM. For each partial transcript it receives, the LLM generates a candidate text response \u2014 called an oracle \u2014 and streams it back to the front-end. Because the user\u2019s speech is still arriving, these oracles start as educated guesses and become progressively more accurate as the transcript grows more complete.<\/p>\n

The front-end S2S transformer then conditions its ongoing speech output on both its own internal context and these incoming oracle tokens. When a new, better oracle arrives, the model can correct course \u2014 effectively updating its response mid-sentence, the way a human might. Because both modules run asynchronously and independently, the initial response latency stays near zero.<\/p>\n

Training on Simulated Oracles<\/strong><\/h3>\n

One challenge is that no naturally occurring dataset contains oracle signals. Sakana AI research team addresses this with a technique called Simulated Oracle Augmentation<\/strong>. Using a \u2018simulator\u2019 LLM and a standard conversational dataset (user utterance + ground-truth response), the research team generates synthetic oracle sequences that mimic what a real-time LLM would produce across different levels of transcript completeness. They define six hint levels (0\u20135), ranging from a completely unguided guess at hint level 0 to the verbatim ground-truth response at hint level 5. The training data for KAME was built from 56,582 synthetic dialogues drawn from MMLU-Pro, GSM8K, and HSSBench, converted to audio via TTS and augmented with these progressive oracle sequences.<\/p>\n

Results: Near-Cascaded Quality, Near-Zero Latency<\/strong><\/h3>\n

Evaluations on a speech-synthesized subset of the MT-Bench multi-turn Q&A benchmark \u2014 specifically the reasoning, STEM, and humanities categories (Coding, Extraction, Math, Roleplay, and Writing were excluded as unsuitable for speech interaction) \u2014 show a dramatic improvement. Moshi alone scores 2.05 on average. KAME with gpt-4.1 as the back-end scores 6.43, and KAME with claude-opus-4-1 as the back-end scores 6.23 \u2014 both at essentially the same latency as Moshi. The leading cascaded system, Unmute (also backed by gpt-4.1), scores 7.70, but with a median latency of 2.1 seconds versus near-zero for KAME.<\/p>\n

To isolate back-end capability from timing effects, the research team also evaluated the back-end LLM\u2019s text responses from the final oracle injection in each KAME session directly \u2014 bypassing the premature-generation problem entirely. Those scores averaged 7.79 (reasoning 6.48, STEM 8.34, humanities 8.56), comparable to Unmute\u2019s 7.70. This confirms that KAME\u2019s gap to cascaded systems is not a ceiling on the back-end LLM\u2019s knowledge, but a consequence of starting to speak before the full user query has been heard.<\/p>\n

Crucially, KAME is fully back-end agnostic<\/strong>. The front-end was trained using gpt-4.1-nano as the primary back-end, but swapping in claude-opus-4-1 or gemini-2.5-flash at inference time requires no retraining. In Sakana AI\u2019s experiments, claude-opus-4-1 tended to outperform gpt-4.1 on reasoning tasks, while gpt-4.1 scored higher on humanities questions \u2014 suggesting practitioners can route queries to the most task-appropriate LLM without touching the front-end model.<\/p>\n

Key Takeaways<\/strong><\/h3>\n