{"id":985,"date":"2026-05-27T06:31:02","date_gmt":"2026-05-26T22:31:02","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=985"},"modified":"2026-05-27T06:31:02","modified_gmt":"2026-05-26T22:31:02","slug":"stability-ai-releases-stable-audio-3-a-family-of-fast-latent-diffusion-models-for-audio-generation-and-editing","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=985","title":{"rendered":"Stability AI Releases Stable Audio 3: A Family of Fast Latent Diffusion Models for Audio Generation and Editing"},"content":{"rendered":"

Stability AI has released open weights for Stable Audio 3 along with a technical research paper<\/a>. Stable Audio 3 is a family of latent diffusion models that generate stereo audio at 44.1 kHz. The models support variable-length outputs, inpainting-based editing, and fast inference.<\/p>\n

What Is Stable Audio 3?<\/strong><\/h2>\n

Stable Audio 3 is a family of three model scales: small, medium, and large. A latent diffusion model generates audio by learning to progressively remove noise from a compressed representation of audio, called a latent. The model learns a mapping from noise to data by training on many (noisy latent, audio) pairs.<\/p>\n

The three model scales differ in capacity and maximum generation length. All parameter counts below are for the diffusion transformer component only. Each model also includes a SAME autoencoder (108M parameters for SAME-S, 852M for SAME-L).<\/p>\n

    \n
  • small-music<\/strong> \u2014 459M diffusion transformer parameters, up to 2 minutes, music only.<\/li>\n
  • small-sfx<\/strong> \u2014 459M diffusion transformer parameters, up to 2 minutes, sound effects only.<\/li>\n
  • medium<\/strong> \u2014 1.4B diffusion transformer parameters, up to 6 minutes and 20 seconds, music and sound effects.<\/li>\n
  • large<\/strong> \u2014 2.7B diffusion transformer parameters, up to 6 minutes and 20 seconds, music and sound effects.<\/li>\n<\/ul>\n

    Open weights for small and medium are available on Hugging Face. Large is available under an enterprise license.<\/p>\n

    Architecture: Two Components<\/strong><\/h2>\n

    Stable Audio 3 has two main components:<\/strong> a semantic-acoustic autoencoder called SAME, and a diffusion transformer that generates latent sequences conditioned on text, duration, and inpainting masks.<\/p>\n

    \n
    \"\"
    https:\/\/arxiv.org\/pdf\/2605.17991<\/figcaption><\/figure>\n<\/div>\n

    The SAME Autoencoder<\/strong><\/h4>\n

    SAME (Semantically-Aligned Music autoEncoder) converts stereo 44.1 kHz audio into a compact latent representation and back. Its key design parameter is a 4096\u00d7 downsampling ratio \u2014 substantially higher than the 1024\u00d7 to 2048\u00d7 ratios common in prior audio autoencoders. This higher ratio reduces latent sequence lengths enough for long-form generation to run on consumer hardware.<\/p>\n

    SAME achieves its 4096\u00d7 compression through two stages. First, a patching stage<\/strong> reshapes stereo audio into non-overlapping patches of 256 samples per channel, achieving 256\u00d7 downsampling. Second, a Transformer Resampling Block (TRB)<\/strong> applies a further 16\u00d7 downsampling using learnable output embeddings interleaved with the input sequence, processed through a transformer. The combined output is a 256-dimensional latent sequence at approximately 10.76 Hz for a 44.1 kHz input.<\/p>\n

    The SAME autoencoder is trained with five loss types<\/strong>: spectral reconstruction, adversarial, diffusion alignment, semantic regression (predicting chroma and interaural level difference), and contrastive latent alignment. These losses push the latent to preserve both acoustic reconstruction quality and semantic structure. A soft-normalisation bottleneck constrains the scale of the latent, providing deterministic encoding.<\/p>\n

    The SAME autoencoder is frozen during diffusion training. Small models use SAME-S (108M parameters, optimized for CPU inference); medium and large use SAME-L (852M parameters).<\/p>\n

    The Diffusion Transformer<\/strong><\/h4>\n

    The diffusion transformer operates on SAME latents. Conditioning enters through three pathways:<\/strong><\/p>\n

      \n
    1. Text<\/strong> \u2014 a frozen T5Gemma encoder produces a sequence of 256 embeddings of dimension 768. Short prompts are padded to 256 with a learned embedding; long prompts are truncated.<\/li>\n
    2. Duration<\/strong> \u2014 encoded as a Fourier features vector and injected via both Adaptive Layer Normalization (AdaLN) and cross-attention alongside the text prompt.<\/li>\n
    3. Inpainting<\/strong> \u2014 a binary mask concatenated with the masked reference audio is projected through a 2-layer MLP and added to the residual stream of each transformer block.<\/li>\n<\/ol>\n

      Each transformer block contains self-attention, cross-attention, local-additive conditioning for inpainting, and a SwiGLU feed-forward network. Medium and large use differential attention<\/strong>, which computes two separate attention maps using two (Q, K) pairs sharing one set of values V, then subtracts one map from the other. This cancels attention patterns that are common to both heads. The transformer prepends 64 learnable memory embeddings<\/strong> before processing each sequence. These provide a global context buffer that every position can attend to, and are removed before computing any loss.<\/p>\n

      Variable-Length Generation<\/strong><\/h2>\n

      Most prior latent diffusion models for audio operate at a fixed maximum sequence length. Generating a short clip still requires running inference at full length, wasting compute on silence. Stable Audio 3 is trained to generate audio at variable lengths natively, using three mechanisms:<\/p>\n

        \n
      • Variable-length flash attention and masked loss<\/strong> \u2014 sequences shorter than the batch maximum are right-padded in latent space. Padding positions are excluded from self-attention and from the loss.<\/li>\n
      • Per-element timestep shifts<\/strong> \u2014 longer sequences retain more structure at a given noise level due to redundancy between neighboring elements. To compensate, the noise schedule is shifted toward higher noise levels for longer sequences during training, using a logistic shift parameterized by \u00b5 (interpolating between \u00b5min=0.5 and \u00b5max=1.15 based on sequence length).<\/li>\n
      • Silence augmentation<\/strong> \u2014 the signal region is randomly extended with pre-computed silence embeddings drawn from an exponential distribution, averaging 4 seconds. This teaches the model to terminate audio with natural silence.<\/li>\n<\/ul>\n

        The practical result is that inference cost scales with output duration. Medium generates 20 seconds of audio in approximately 0.62 seconds on an H200. Generating 380 seconds takes 1.31 seconds on the same hardware.<\/p>\n

        Three-Stage Training Pipeline<\/strong><\/h2>\n

        Stage 1 \u2014 Flow Matching Pre-Training.<\/strong> The model learns a velocity field that transports Gaussian noise toward audio latents. Training uses minibatch optimal transport coupling<\/strong> via Sinkhorn iterations, which pairs each data sample with the closest available noise vector in the batch. This straightens training trajectories and reduces crossing transport paths. Inpainting is trained jointly throughout: at each step, one of three mask types is sampled \u2014 full mask (80%, equivalent to unconditional generation), random segment masks (10%), or a causal prefix mask for continuation (10%).<\/p>\n

        Stage 2 \u2014 Distillation Warmup.<\/strong> A frozen copy of the flow matching model (teacher) generates 15-step DPM++ trajectories with CFG scale 5. The student is trained for 10,000 steps to map any intermediate noisy state directly to the teacher\u2019s final denoised output in one step, using an MSE loss. This collapses the multi-step ODE into a single-step denoiser. The trade-off is that MSE regression produces outputs that regress toward the conditional mean, reducing fine-grained detail.<\/p>\n

        Stage 3 \u2014 Adversarial Post-Training.<\/strong> This stage replaces the MSE objective with a relativistic adversarial setup. A discriminator (initialized from the base flow matching model) evaluates the student\u2019s one-step denoised outputs directly against real data. The teacher is discarded entirely at this stage. The generator is trained with two losses: a relativistic adversarial loss (L_R) and a CLAP alignment loss (L_CLAP). The discriminator is trained with L_R and a contrastive loss (L_C) that penalizes the discriminator for ignoring text-audio alignment (it is trained to distinguish correctly paired audio-text pairs from shuffled ones). The adversarial setup allows the model to recover the perceptual sharpness that MSE distillation removes.<\/p>\n

        Inference: Ping-Pong Sampling and No CFG<\/strong><\/h2>\n

        The post-trained model can generate audio in a single forward pass. However, single-step generation from pure noise remains difficult. Stable Audio 3 uses ping-pong sampling<\/strong> at inference: the model denoises to a clean estimate, then adds new noise at a reduced level, then denoises again. This repeats for 8 steps using a logSNR-uniform schedule (N+1 equally-spaced steps in the interval [\u03bbmin, \u03bbmax] = [\u22126.2, 2.0]). The iterative denoise-then-renoise schedule allows each step to correct errors from the previous step.<\/p>\n

        Stable Audio 3 does not require classifier-free guidance (CFG) at inference<\/strong>. Standard diffusion models run two forward passes per step \u2014 one conditional, one unconditional \u2014 and interpolate. Here, CFG quality gains are internalized during distillation warmup, where the student is trained to match CFG-enhanced teacher trajectories. Text-audio alignment is further reinforced through L_CLAP during adversarial post-training. This eliminates the two-pass-per-step cost of CFG.<\/p>\n

        Prompt formatting note:<\/strong> All Stable Audio 3 models trained on AudioSparx (small-music, medium, large) require prompt prefixes to function correctly. Music prompts should be prepended with \"TrackType: Music, VocalType: Instrumental,\"<\/code> and sound effects prompts with \"TrackType: SFX,\"<\/code>. <\/p>\n

        Evaluation Results<\/strong><\/h2>\n

        Instrumental music (Song Describer Dataset, 120s).<\/strong> On FAD (lower is better) and CLAP score (higher is better), large achieves FAD 0.101 \/ CLAP 0.393. Medium achieves FAD 0.107 \/ CLAP 0.390. Stable Audio 2.5 (the internal prior-generation baseline) achieves FAD 0.106 \/ CLAP 0.395. In the listening test, medium and large score higher on musicality (MUS) than Stable Audio 2.5 (4.15 and 4.30 vs. 3.70 out of 5, respectively). Inference time for 120s audio on an H200: 0.45s for small, 0.78s for medium, 0.81s for large. Stable Audio 2.5 takes 0.85s for the same length.<\/p>\n

        Sound effects (BBC Sound Effects Dataset, 5s).<\/strong> Medium achieves FAD 0.369 \/ CLAP 0.369. The next-best open-weight baselines are Stable Audio Open Small (FAD 0.500 \/ CLAP 0.277) and Stable Audio Open (FAD 0.501 \/ CLAP 0.263). Woosh Flow scores FAD 0.580.<\/p>\n

        Audio editing (inpainting).<\/strong> The research team evaluates three inpainting settings: single region, two independent regions, and continuation. For music, medium achieves FAD-full of 0.046 on single inpainting and 0.046 on double inpainting. Large achieves 0.047 on both. For continuation, medium achieves FAD-full 0.074 and large achieves 0.071. Sound effects results follow a similar pattern; continuation shows higher FAD than inpainting in both domains, which the team attributes to the model having less surrounding audio context to anchor the generation.<\/p>\n

        Comparison<\/strong><\/h2>\n
        \n
        \n
        Model specs<\/div>\n
        Music benchmarks (SDD, 120s)<\/div>\n
        SFX benchmarks (BBC, 5s)<\/div>\n<\/div>\n
        \n
        \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
        Model<\/th>\nDeveloper<\/th>\nReleased<\/th>\nArchitecture<\/th>\nParameters<\/th>\nMax length<\/th>\nSample rate<\/th>\nDomain<\/th>\nOpen weights<\/th>\nInpainting<\/th>\n<\/tr>\n<\/thead>\n
        STABLE AUDIO LINEAGE<\/td>\n<\/tr>\n
        Stable Audio Open<\/td>\nStability AI<\/td>\nJul 2024<\/td>\nLatent diffusion (DiT)<\/td>\nDiT 1057M + AE 156M + T5 109M<\/td>\n47s<\/td>\n44.1kHz stereo<\/td>\nMusic + SFX<\/td>\nYes<\/span><\/td>\nNo<\/td>\n<\/tr>\n
        Stable Audio Open Small<\/td>\nStability AI<\/td>\n2024<\/td>\nLatent diffusion (DiT)<\/td>\nNot published<\/td>\n11s<\/td>\n44.1kHz stereo<\/td>\nSFX<\/td>\nYes<\/span><\/td>\nNo<\/td>\n<\/tr>\n
        Stable Audio 2.5<\/td>\nStability AI<\/td>\nInternal<\/td>\nLatent diffusion (DiT)<\/td>\nNot published<\/td>\n190s (3m 10s)<\/td>\n44.1kHz stereo<\/td>\nMusic<\/td>\nNot released<\/span><\/td>\nNo<\/td>\n<\/tr>\n
        SA3 small-music \u2605<\/b><\/td>\nStability AI<\/td>\nMay 2026<\/td>\nLatent diffusion (SAME + DiT)<\/td>\nDT 459M + SAME-S 108M<\/td>\n2m<\/td>\n44.1kHz stereo<\/td>\nMusic only<\/td>\nYes<\/span><\/td>\nYes<\/td>\n<\/tr>\n
        SA3 small-sfx \u2605<\/b><\/td>\nStability AI<\/td>\nMay 2026<\/td>\nLatent diffusion (SAME + DiT)<\/td>\nDT 459M + SAME-S 108M<\/td>\n2m<\/td>\n44.1kHz stereo<\/td>\nSFX only<\/td>\nYes<\/span><\/td>\nYes<\/td>\n<\/tr>\n
        SA3 medium \u2605<\/b><\/td>\nStability AI<\/td>\nMay 2026<\/td>\nLatent diffusion (SAME + DiT)<\/td>\nDT 1.4B + SAME-L 852M<\/td>\n6m 20s<\/td>\n44.1kHz stereo<\/td>\nMusic + SFX<\/td>\nYes<\/span><\/td>\nYes<\/td>\n<\/tr>\n
        SA3 large \u2605<\/b><\/td>\nStability AI<\/td>\nMay 2026<\/td>\nLatent diffusion (SAME + DiT)<\/td>\nDT 2.7B + SAME-L 852M<\/td>\n6m 20s<\/td>\n44.1kHz stereo<\/td>\nMusic + SFX<\/td>\nEnterprise<\/span><\/td>\nYes<\/td>\n<\/tr>\n
        COMPETITORS<\/td>\n<\/tr>\n
        TangoFlux<\/td>\nSUTD \/ NVIDIA \/ Lambda<\/td>\nDec 2024<\/td>\nFlow matching (DiT + MMDiT)<\/td>\n515M<\/td>\n30s<\/td>\n44.1kHz<\/td>\nSFX<\/td>\nYes (Apache 2.0)<\/span><\/td>\nNo<\/td>\n<\/tr>\n
        Woosh Flow<\/td>\nSony AI<\/td>\nApr 2026<\/td>\nFlow matching<\/td>\nNot published<\/td>\n5s<\/td>\nNot disclosed<\/td>\nSFX<\/td>\nYes (MIT)<\/span><\/td>\nNo<\/td>\n<\/tr>\n
        Woosh DFlow<\/td>\nSony AI<\/td>\nApr 2026<\/td>\nDistilled flow matching<\/td>\nNot published<\/td>\n5s<\/td>\nNot disclosed<\/td>\nSFX<\/td>\nYes (MIT)<\/span><\/td>\nNo<\/td>\n<\/tr>\n
        DiffRhythm 2<\/td>\nASLP Lab (NPU)<\/td>\nOct 2025<\/td>\nBlock flow matching (semi-autoregressive)<\/td>\nNot published<\/td>\n210s (3m 30s)<\/td>\n48kHz output<\/td>\nMusic + vocals<\/td>\nYes<\/span><\/td>\nNo<\/td>\n<\/tr>\n
        ACE-Step 1.5<\/td>\nACE Studio \/ StepFun<\/td>\nJan 2026<\/td>\nHybrid LM (0.6B\u20134B) + DiT (up to 4B)<\/td>\nLM 0.6B\u20134B + XL DiT 4B<\/td>\n10m<\/td>\nNot disclosed<\/td>\nMusic + vocals + lyrics<\/td>\nYes<\/span><\/td>\nNo<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/div>\n

        \n \u2605 SA3 rows:<\/b> Parameter counts are for the diffusion transformer (DT) component only; SAME autoencoder params are listed separately. Total model size including SAME: small ~567M, medium ~2.25B, large ~3.55B.
        \n Stable Audio 2.5<\/b> is an internal Stability AI model not publicly released; included as prior-generation internal baseline from the SA3 paper.
        \n DiffRhythm 2<\/b> VAE processes 24kHz input audio and reconstructs at 48kHz (arXiv:2510.22950).\n <\/p>\n<\/div>\n

        \n

        Evaluation setup:<\/b> Song Describer Dataset (SDD), 120s instrumental music generations, H200 GPU. FAD uses LAION-CLAP embeddings (630k-audioset-best.pt). OVL\/REL\/MUS are mean opinion scores (1\u20135) from a 14-participant listening test. Source: SA3 paper Tables 3 and 4. Bold + underline<\/b> = best score in column.<\/p>\n

        \n\n\n\n\n\n\n\n\n\n\n\n\n\n
        Model<\/th>\nFAD \u2193<\/th>\nCLAP \u2191<\/th>\nOVL \u2191 (1\u20135)<\/th>\nREL \u2191 (1\u20135)<\/th>\nMUS \u2191 (1\u20135)<\/th>\nInference (H200)<\/th>\nSampler \/ steps<\/th>\n<\/tr>\n<\/thead>\n
        COMPETITORS<\/td>\n<\/tr>\n
        DiffRhythm 2<\/td>\n0.293<\/td>\n0.158<\/td>\n3.05 \u00b1 0.94<\/td>\n2.10 \u00b1 1.29<\/td>\n2.60 \u00b1 1.10<\/td>\n3.88s<\/td>\n\u2014<\/td>\n<\/tr>\n
        ACE-Step 1.5 xl-turbo<\/td>\n0.193<\/td>\n0.321<\/td>\n3.35 \u00b1 1.09<\/td>\n3.30 \u00b1 1.13<\/td>\n3.15 \u00b1 1.31<\/td>\n6.23s<\/td>\n\u2014<\/td>\n<\/tr>\n
        STABILITY AI \u2014 PRIOR GENERATION<\/td>\n<\/tr>\n
        Stable Audio 2.5 (internal)<\/td>\n0.106<\/td>\n0.395<\/td>\n3.90 \u00b1 0.79<\/td>\n4.30 \u00b1 0.66<\/td>\n3.70 \u00b1 0.92<\/td>\n0.85s<\/td>\nDPM++ 3M SDE, 8 steps, CFG 6<\/td>\n<\/tr>\n
        STABLE AUDIO 3 \u2014 POST-TRAINED (8 PING-PONG STEPS, NO CFG)<\/td>\n<\/tr>\n
        SA3 small-music<\/b><\/td>\n0.145<\/td>\n0.393<\/td>\n3.20 \u00b1 0.89<\/td>\n3.60 \u00b1 0.94<\/td>\n3.15 \u00b1 0.81<\/td>\n0.45s<\/td>\nPingPong, 8 steps<\/td>\n<\/tr>\n
        SA3 medium<\/b><\/td>\n0.107<\/td>\n0.390<\/td>\n4.20 \u00b1 0.89<\/td>\n4.25 \u00b1 0.85<\/td>\n4.15 \u00b1 0.93<\/td>\n0.78s<\/td>\nPingPong, 8 steps<\/td>\n<\/tr>\n
        SA3 large<\/b><\/td>\n0.101<\/td>\n0.393<\/td>\n3.95 \u00b1 0.89<\/td>\n3.80 \u00b1 1.11<\/td>\n4.30 \u00b1 0.73<\/td>\n0.81s<\/td>\nPingPong, 8 steps<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/div>\n

        \n FAD:<\/b> Fr\u00e9chet Audio Distance \u2014 lower is better. CLAP:<\/b> cosine similarity between text and audio embeddings \u2014 higher is better.
        \n OVL<\/b> = overall production quality. REL<\/b> = text relevance. MUS<\/b> = musicality (melody\/harmony coherence).
        \n ACE-Step 1.5 and DiffRhythm 2 evaluated with instrumental prompts only for fair comparison with SA3 (instrumental-only models). SA3 base flow matching models (50 steps, CFG 7, Euler sampler) are not shown here; see SA3 paper Table 11 for that comparison.\n <\/p>\n<\/div>\n

        \n

        Evaluation setup:<\/b> BBC Sound Effects Dataset, \u22645s generations matched to reference duration, H200 GPU. FAD uses LAION-CLAP embeddings. OVL\/REL from 14-participant listening test. Source: SA3 paper Table 5. Bold + underline<\/b> = best score in column.<\/p>\n

        \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
        Model<\/th>\nFAD \u2193<\/th>\nCLAP \u2191<\/th>\nOVL \u2191 (1\u20135)<\/th>\nREL \u2191 (1\u20135)<\/th>\nInference (H200)<\/th>\nSampler \/ steps<\/th>\n<\/tr>\n<\/thead>\n
        COMPETITORS<\/td>\n<\/tr>\n
        TangoFlux<\/td>\n0.760<\/td>\n0.179<\/td>\n2.35 \u00b1 1.04<\/td>\n3.25 \u00b1 1.37<\/td>\n1.90s<\/td>\nFlow matching, 50 steps, CFG 4.5<\/td>\n<\/tr>\n
        Woosh DFlow<\/td>\n0.619<\/td>\n0.228<\/td>\n3.10 \u00b1 1.25<\/td>\n3.20 \u00b1 1.64<\/td>\n0.06s<\/td>\nDistilled flow, 4 steps<\/td>\n<\/tr>\n
        Woosh Flow<\/td>\n0.580<\/td>\n0.277<\/td>\n3.45 \u00b1 1.19<\/td>\n3.80 \u00b1 1.28<\/td>\n1.92s<\/td>\nAdaptive ODE (~72 steps avg)<\/td>\n<\/tr>\n
        STABILITY AI \u2014 PRIOR GENERATION<\/td>\n<\/tr>\n
        Stable Audio Open<\/td>\n0.501<\/td>\n0.263<\/td>\n2.95 \u00b1 1.32<\/td>\n3.30 \u00b1 1.30<\/td>\n12.30s<\/td>\nDPM++ 3M SDE, 100 steps, CFG 7<\/td>\n<\/tr>\n
        Stable Audio Open Small<\/td>\n0.500<\/td>\n0.277<\/td>\n3.10 \u00b1 1.12<\/td>\n3.55 \u00b1 1.00<\/td>\n0.24s<\/td>\nPingPong, 8 steps<\/td>\n<\/tr>\n
        STABLE AUDIO 3 \u2014 POST-TRAINED (8 PING-PONG STEPS, NO CFG)<\/td>\n<\/tr>\n
        SA3 small-sfx<\/b><\/td>\n0.395<\/td>\n0.351<\/td>\n3.35 \u00b1 1.39<\/td>\n3.25 \u00b1 1.45<\/td>\n0.41s<\/td>\nPingPong, 8 steps<\/td>\n<\/tr>\n
        SA3 medium<\/b><\/td>\n0.369<\/td>\n0.369<\/td>\n3.65 \u00b1 1.14<\/td>\n3.95 \u00b1 1.23<\/td>\n0.60s<\/td>\nPingPong, 8 steps<\/td>\n<\/tr>\n
        SA3 large<\/b><\/td>\n0.358<\/td>\n0.370<\/td>\n3.60 \u00b1 0.94<\/td>\n3.85 \u00b1 1.04<\/td>\n0.64s<\/td>\nPingPong, 8 steps<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/div>\n

        \n Woosh DFlow achieves the fastest inference (0.06s) but at a quality cost \u2014 higher FAD than Woosh Flow. SA3 small-sfx, medium, and large all outperform every competitor on FAD and CLAP at the 5s generation length.
        \n SA3 models do not use classifier-free guidance (CFG) at inference. CFG quality gains are internalized during distillation warmup training.\n <\/p>\n<\/div>\n<\/div>\n

        Key Takeaways<\/strong><\/h2>\n