Apple's tri-modal masked diffusion model trains from scratch on text, image, audio

Apple researchers introduced the first tri-modal Masked Diffusion Model (MDM) pretrained from scratch on text, image-text, and audio-text data. The work, published ahead of NeurIPS 2026, systematically analyzes multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects, and provides optimized inference sampling defaults.

The model diverges from prior approaches that initialized and finetuned a base unimodal model for bi-modal generation. Instead, the Apple team pretrained a unified model across three modalities from scratch. A preliminary model at 3B scale, trained on 6.4T tokens, showed strong results in text generation, text-to-image (T2I), and text-to-speech (T2S) tasks, according to the paper.

Batch-size reparameterization

A key contribution is a novel stochastic differential equation (SDE) based reparameterization that eliminates the need to tune the optimal batch size, a requirement reported in recent work. The reparameterization decouples the physical batch size, chosen for compute efficiency such as GPU saturation, FLOP-efficiency, and wall-clock time, from the logical batch size, which balances gradient variance during stochastic optimization.

For inference engineers, this distinction matters when serving models on fixed hardware. The physical batch size can be set to maximize throughput on a given accelerator without compromising training dynamics, while the logical batch size controls optimization behavior. The decoupling suggests that serving configurations can be tuned independently of the training setup.

Scaling analysis and inference defaults

The study claims to be the largest scale systematic open study of multimodal discrete diffusion models to date. The analysis covers scaling behaviors across modalities, including how model performance scales with compute and data. The paper provides optimized inference sampling defaults, which are directly applicable to deployment scenarios where latency and cost constraints dominate.

The authors include researchers from Apple, the University of Cambridge, MIT, and Google DeepMind, with some work done while at Apple.

For teams serving open-weight diffusion models, the tri-modal approach suggests a path toward a single model handling text, image, and audio generation tasks, potentially reducing the need for multiple specialized models. The scaling law analysis offers guidance on data mixing ratios and noise schedules, which can inform how to adapt the model to specific workloads or fine-tune it for downstream tasks.

The practical implications for edge AI and inference serving remain to be validated by the broader community.