Meta's Muse Glimmer 30B Model Runs On Device via ExecuTorch

Meta introduced Muse Glimmer, an open-weight 30-billion-parameter model distilled from Muse Spark for on-device agentic workflows. ExecuTorch now supports running Muse Glimmer end-to-end on NVIDIA GPUs and Macs with Apple silicon, with prebuilt PTE artifact bundles available for download from Hugging Face.

What ExecuTorch Does Differently

Most local AI frameworks rewrite models in non-Python languages, which worked when LLMs were standard text transformers. Today's models are more complex, with novel architectures, multimodal inputs and outputs, and advanced decoding algorithms like DFlash, a parallel diffusion-based speculative decoding method for low latency. Reimplementing these across backends does not scale.

ExecuTorch lets engineers implement the model and its decoding strategy in PyTorch, then export to ExecuTorch for deployment. The framework handles backend-specific lowering, including Triton on CUDA and MLX-native or custom Metal on Apple silicon. Ahead-of-time compilation optimizes the full execution path end-to-end, not just individual ops.

Running Muse Glimmer

Prebuilt PTEs are published for NVIDIA CUDA and Apple Silicon (Metal), covering text-only and text-plus-image artifacts, with and without DFlash speculative decoding. To build custom artifacts, users can follow the ExecuTorch Muse Glimmer README and select backend, modality, context length, and DFlash usage. CUDA export compiles and autotunes Triton kernels for the detected GPU architecture, and exporting on the same GPU architecture that will run the artifact gives the best results.

The runtime builds via CMake presets for CUDA and MLX backends, producing solo_runner, dflash_runner, and a serving worker. The example llm_server supports agentic use cases. A standalone command-line inference example runs a vision model with DFlash, and an agent server example integrates with the Pi coding agent, which can create files, run tests, and interact with the user.

Performance and Optimizations

Meta reports that on an M5 Pro with 64 GiB, the solo runner achieves 21.6 tokens per second on a text-image input, while DFlash reaches 33.0 tokens per second, a 52.8% improvement without quality regression. Performance charts show prefill and decode throughput on NVIDIA A100 and Apple M5-max with and without DFlash.

Under the hood, ExecuTorch enables DFlash by optimizing target and draft interoperability through weight sharing, exporting both into a single PTE. The DFlash block dimension is exported dynamically, allowing one PTE to support runtime-selectable block lengths. The runtime supports both greedy decoding and rejection sampling.

GGUF loading and k-quant support are built in. Q4_K, Q5_K, and Q6_K weights map to packed INT4, INT5, and INT6 with dp4a GEMV kernels on CUDA, and to repacked or fused Metal kernels on MLX. On MLX, adjacent sub-blocks with identical scale and min are merged into larger group sizes up to 128 when lossless.

For agentic workloads, one model load serves multiple isolated conversations via per-session mutable-state rebinding. Harmony chat templating with reasoning routing is included, as is a parser for the model's XML tool-call format, supporting multiple calls in one turn.

CUDA optimizations capture decode into a CUDA graph, reducing per-kernel launch overhead to one submission. Packed K-quant kernels accelerate low-batch decode, and length-aware split-K FlashDecoding++ paths optimize single-token decode and small DFlash verification blocks. On MLX, RMSNorm, RoPE, SDPA, KV-cache updates, and quantized linear operations use MLX-native or custom Metal implementations.

Muse Glimmer supports a 128K+ token context with an efficient KV-cache: only 13 of its 52 layers are global, while the other 39 use sliding windows. ExecuTorch supports this efficiently, making long-context use cases practical on edge devices.

Current Limits and Next Steps

This initial release supports text and image inputs; video input is not yet supported. There is no cross-session prefix sharing, checkpointing, or continuous batching yet, but these are actively being worked on to improve agentic workflow support.