Three Tests to Run Before You Switch from LoRA to FullFT

Fireworks AI ran controlled experiments on Qwen3.5-9B to isolate why LoRA sometimes trails Full Parameter Fine-Tuning, or FullFT. The company's blog post, published July 30, 2026, describes three tests that separate data coverage, optimization, and adapter capacity as possible causes of a quality gap. The tests use three synthetic tasks with automatic checkers, so every answer is scored without a human judge.

The first test, placement, arranges components on a line to minimize wiring while obeying hard constraints. A single violation invalidates the answer. On a 2,000-row dataset, LoRA at rank 32 and rank 128 both scored around 34% validity, while FullFT reached 57%. Increasing the data to 20,000 rows roughly doubled LoRA validity and cut the FullFT gap in half, but quadrupling the rank moved the result by one point. A learning-rate sweep then closed the gap entirely: with a peak rate of 8e-5, rank-32 LoRA reached 82.67% validity versus 83.00% for FullFT on a common suite. The 0.33-point difference had a 95% confidence interval of −4.33 to +5.00 points, so the evaluation could not distinguish the two methods. The verdict: the adapter was not too small, the gap came from data and learning rate.

The second test, register allocation, assigns processor registers so values in use at the same time never share one. Here rank moved the result. LoRA at rank 8 achieved 80.50% valid allocations, rank 32 reached 82.50%, and rank 128 hit 90.50%, close to FullFT's 93.00%. Exact decision sequence also improved with rank, from 3.50% at rank 8 to 25.50% at rank 128, though FullFT retained a lead at 35.50%. The company notes that exact sequence measures imitation, not practical quality. A separate 60-graph control reproduced the direction, but the sample was too small to estimate the effect precisely, and the data and learning-rate checks remain open.

The third test, Nexa VM, executes programs in a synthetic instruction language and reports every intermediate state. An early 10,000-row comparison showed FullFT leading LoRA by 20 points on final answers and 31 points on exact traces. A learning-rate sweep reversed that: with tuned rates, LoRA at rank 64 reached 98.44% final-answer accuracy versus 96.88% for FullFT on a 64-example depth-8 subset. Across the full 480-example suite, LoRA and FullFT scored 478 versus 476 correct final answers. The company's verdict: a 31-point gap that looked like FullFT learning faster was really an untuned learning rate.

On supported behavior, a rank sweep from 64 down to 1 kept exact-trace accuracy between 99.17% and 100%, so rank did not separate the tested adapters. The failure mode appeared on longer programs: models trained on a fixed range of register positions became less reliable when depth-16 programs introduced new positions. Increasing rank did not produce a monotonic recovery, but randomizing register starts during training did.

The company's protocol is a per-gap ladder: tune the learning rate first, then broaden data coverage, then increase rank. The results align with prior work from Biderman et al. and Thinking Machines, which reported that LoRA can match FullFT with a well-configured recipe and a learning rate roughly an order of magnitude above FullFT's. In these experiments, the best LoRA rate was 8e-5, far above FullFT's best rate of 1e-6 on Nexa.