J-Lens readouts reproduce on Kimi K3 and Qwen3.5-9B, showing pre-token signals

Fireworks AI applied Anthropic's Jacobian Lens, or J-Lens, to two open models, Kimi K3 and Qwen3.5-9B, and found that task-related vocabulary appears in the models' hidden states before it shows up in the generated text. The probe, a classifier fitted on internal states at specific token positions and layers, produces a ranked list of tokens that the model is leaning toward. The results suggest that open models carry readable silent signals before they write a single token.

Reproducing the paired copy test

Fireworks first ran a paired copy experiment on Kimi. The model was asked to echo the sentence "The old painting hung crookedly on the wall." twice, once with an arithmetic focus instruction and once with a citrus focus instruction. Both times Kimi produced the exact same visible output, but the lens readout showed arithmetic-related tokens like 7, 9, and "answer" in the arithmetic condition, and citrus terms like orange, lemon, and lime in the citrus condition. Neither run leaked into the other, meaning arithmetic tokens never cracked the top 10 during the citrus run and vice versa.

Across ten Kimi examples, each of the 20 reviewed word lists, two per example, had at least one listed word among the readout's top 10 results. In one episode, the intermediate values 21 and 42 appeared in the lens readout before either number was written in the completion. In a spider-leg riddle, the word "spider" read out before the model named the bridge entity. In a rhyming couplet task, several rhyme candidates appeared before the first exact match.

Transfer to Qwen3.5-9B

The same 20 word lists, selected using Kimi readouts, were applied unchanged to a separately fitted Qwen3.5-9B lens. Qwen used its own tokenizer, a separately fitted lens, and deterministic native completions with thinking disabled. Nineteen of the 20 word lists met the found criteria, meaning at least one listed word appeared among the top 10 matched results at a sampled position and model depth. Specific examples included arithmetic values like 7 and 9 co-occurring with process terms like evaluate and subtract, and citrus completions mentioning lemon zest and orange peel.

Where the signals live

A second test split each transcript into three phases: the user message prefill, the chat template prefill, and the eligible response rollout prefix before the concept family's first explicit appearance. This test used 40 additional word lists and retained the top 50 results. Kimi hit 40/40 on user-message regions, 40/40 on chat-template regions, and 24/24 on eligible response-prefix regions. Qwen was nearly identical at 39/40, 39/40, and 23/24. The pattern holds not just during active generation but also while the model is digesting the prompt.

What it takes to run this

The lens was fitted on 14 task-independent calibration passages covering topics like metal heat treatment, linguistic borrowing, crop rotation, and traditional navigation. These passages were separate from the ten holdout user prompts used to examine the model's inner activations. Kimi processes all 39 tokens of a calibration passage normally, but the estimator uses only 22 positions, excluding a predeclared 16-position starting window and the final position. Those 22 positions produce one passage-level estimate at each sampled layer, and the 14 estimates are averaged into a fitted lens.

For inference engineers, the practical takeaway is that hidden states carry recoverable task-related information before output. This could enable debugging of model behavior, monitoring of task adherence, or even steering of generation based on internal readouts. Fireworks AI notes at least one place where the method breaks: a tokenizer problem. The transfer from Kimi to Qwen suggests the approach is not architecture-specific, which matters for teams serving multiple open-weight models.