{"id":1029,"date":"2026-06-04T16:11:05","date_gmt":"2026-06-04T08:11:05","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=1029"},"modified":"2026-06-04T16:11:05","modified_gmt":"2026-06-04T08:11:05","slug":"miso-labs-releases-misotts-an-8b-emotive-text-to-speech-model-with-open-weights","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=1029","title":{"rendered":"Miso Labs Releases MisoTTS: An 8B Emotive Text-to-Speech Model with Open Weights"},"content":{"rendered":"

Miso Labs has released MisoTTS, an open-weights 8-billion-parameter text-to-speech model. It generates expressive speech from both text and audio context. The model uses residual vector quantization (RVQ) to widen its sonic range. This avoids scaling a single flat vocabulary while keeping parameter count fixed. <\/p>\n

What is MisoTTS<\/strong><\/h2>\n

MisoTTS is an 8B-parameter text-to-dialogue RVQ Transformer. It is inspired by the Sesame CSM architecture. It pairs a Llama 3.2-style backbone with a smaller audio decoder. It generates Mimi audio codes from text and optional audio context. The model conditions on both text and prior audio. That second input lets it respond to the speaker\u2019s tone.<\/p>\n

The text vocabulary is 128,256 tokens, and there are 32 audio codebooks. Mimi is the audio tokenizer, and max sequence length is 2,048. Default inference runs in torch.bfloat16<\/code>.<\/p>\n

Miso Labs claims 110ms latency. It lists ElevenLabs at 700ms and Sesame at 300ms. <\/p>\n

The Vocabulary Size Problem<\/strong><\/h2>\n

Standard transformers generate from a fixed vocabulary of discrete tokens. That works when a small vocabulary covers the target space. Human speech does not fit that assumption. It varies across pitch, rhythm, emphasis, emotion, and accent.<\/p>\n

Expanding the audio vocabulary is the obvious fix. But larger vocabularies need more parameters in a standard transformer. Each token must be represented and predicted by the model. Miso Labs calls this the vocabulary size problem.<\/p>\n

The second issue is conditioning. Most TTS models condition only on text. They ignore the interlocutor\u2019s tone. Miso Labs argues this contributes to the \u201cuncanny valley\u201d effect.<\/p>\n

Residual Vector Quantization: The Core Idea<\/strong><\/h2>\n

MisoTTS addresses both problems with residual vector quantization (RVQ). Miso Labs traces RVQ to image-generation research and to Sesame\u2019s CSM for audio. Instead of one token index, the model emits a vector of indices.<\/p>\n

Each audio token is 32 codebook indices over 2048-way codebooks. The model keeps a separate codebook for each position in the vector. To recover the sound, it sums the looked-up vectors. Each codebook adds another refinement to the signal.<\/p>\n

This is what makes the scaling work. Addressable vocabulary equals codebook size raised to the depth. Growing the depth adds no parameters to the model. So MisoTTS reaches about 204832<\/sup>, or roughly 10105<\/sup> addressable tokens. Miso Labs notes naive scaling would require a far larger network.<\/p>\n

\n
\"\"
https:\/\/www.misolabs.ai\/blog\/miso-tts-8b<\/figcaption><\/figure>\n<\/div>\n

The Two-Transformer Architecture<\/strong><\/h2>\n

The model splits into a backbone and a decoder. The backbone is a 7.7B-parameter transformer, autoregressive over time. It predicts the first codebook index and a final hidden state.<\/p>\n

A 300M-parameter decoder then runs autoregressively over depth. It predicts the remaining codebook indices, one position at a time. Each prediction conditions on the indices already chosen in the frame. The same 300M parameters are reused for every position.<\/p>\n

Embeddings follow the same logic. Text tokens use a single lookup. An audio token\u2019s embedding is the sum of per-position codebook lookups. Interleaving text and audio lets the backbone use conversation history. That is how it carries context across turns.<\/p>\n

Strengths and Challenges<\/strong><\/h2>\n

Strengths:<\/strong><\/h4>\n