NVIDIA Jetson Runs 2026 Reasoning Models With NVFP4 and Speculative Decoding

NVIDIA published a guide on September 4, 2026 showing how compact open models released that year can run reasoning and agentic workloads locally on Jetson hardware. The company used Nemotron 3.5 Lightning and Qwen3.8-27B as examples, and reported that combining NVFP4 quantization with speculative decoding delivers up to 6.28x decode throughput speedup over BF16 on Jetson AGX Thor and Jetson AGX Orin.

The two models differ in architecture, and NVIDIA says that choice matters for long-running agents. Nemotron 3.5 Lightning uses a mixture-of-experts design with 30 billion total parameters but activates only 3 billion per token. Qwen3.8-27B is dense, activating all 27 billion parameters each token. NVIDIA recommends Nemotron 3.5 Lightning for response-heavy workflows where faster token generation shortens the overall process, and Qwen3.8-27B for tasks requiring fewer, harder decisions where the agent can spend more time generating each response. For Jetson Orin Nano, the company points to Gemma 4 E4B as a strong starting point.

Optimization techniques

NVIDIA describes two complementary techniques. NVFP4 quantization reduces the data the GPU must move and process during each decoding pass, improving generation speed and cutting memory use while keeping quality close to BF16. Speculative decoding uses a smaller draft model to propose several tokens, which the main model verifies together, advancing generation by multiple tokens per verification step.

Three draft methods run on Jetson: MTP, DFlash, and DSpark. MTP uses prediction heads trained with the main model. DFlash uses a separate diffusion-based draft model to propose a block of tokens in parallel and supports the widest selection of compatible draft checkpoints. DSpark builds on DFlash by correcting drafts and stopping weak proposals early, and can be faster when a matching checkpoint exists but supports fewer checkpoints.

The fastest configuration differed by model. Nemotron 3.5 Lightning performed best with DSpark, while Qwen3.8-27B performed best with DFlash2. NVIDIA advises testing the method and draft checkpoint with the target model rather than assuming one configuration works for every model.

Deployment and validation

NVIDIA provides vLLM commands for both models. For Nemotron 3.5 Lightning, the fastest tested configuration combines NVFP4 with DSpark, running on Jetson AGX Thor or Jetson AGX Orin with a vllm/vllm-openai:v0.28.0 container. For Qwen3.8-27B, the fastest tested configuration combines NVFP4 with DFlash2, using a Triton decode kernel and the same container. Both setups require JetPack 7.2 with NVIDIA Container Runtime and Docker, plus enough storage for model and draft checkpoints.

NVIDIA stresses application-level validation because throughput varies by workload category. The company kept the fastest configuration for each model fixed and tested four SpeedBench categories: writing, reasoning, summarization, and retrieval-augmented generation. RAG and writing benefited most for both models. General benchmarks cannot confirm whether a checkpoint preserves behavior critical to specific data, so developers should test with representative prompts from their own application.