Research · November 17, 2026

Thinking in vectors instead of tokens: what failed first

A frozen 2B model solved 100 of 100 word problems when it thought in text and 54 when it answered directly. On a separate 500-problem set, one vector step in place of the thinking tokens added 8 points over a direct answer and stayed about 60 below text.

Mendrika Ramarlina · November 17, 2026 · 9 min read

A frozen 2B-parameter open language model answered 54 of 100 grade-school word problems when it had to pick an answer immediately. With its text thinking turned on, it answered 100 of 100.

The first stretch of this research tried to keep that computation and drop the tokens: let the model think in continuous vectors instead of writing its reasoning out. Nothing we built in those four days came close. The best result was one vector step through the frozen model, which added 8 points on 500 problems and stayed about 60 points below text thinking.

Cortex, the system the later parts of this series describe, came out of these failures. In its current form a frozen language model encodes text, a small recurrent latent policy does the reasoning over a memory of facts and results, and exact tools (a calculator) do exact computation. Part 1 has the overview. This is research at Saina, not a product.

All the problems here are synthetic multiple-choice word problems from ten elementary templates, with four options, so chance is 25%.

The computation was already in the model

Our first guess for the weak direct answers was comprehension.

We gave the model four recognition probes per problem plus the solve question. It passed 355 of 400 probes and solved 54 of 100. On 28 problems it passed every probe and still got the answer wrong. Recognition probes are not a full test of understanding, but this ruled out the simplest comprehension story.

So we tried to hand it the missing pieces:

Intervention Solved /100
Direct answer, thinking off 54
Comprehension answers supplied 50
Broken into small questions, own intermediate answers 51
Broken into small questions, correct intermediate answers supplied 51
Text thinking on 100

Neither supplied information nor a decomposition helped. Thinking did.

On problems given as plain expressions, the gap showed up again: a generated answer letter went from 55 to 100 with thinking on, and a generated number from 91 to 100.

Our read: the model can do this arithmetic when it has room to compute. The goal became replacing the thinking tokens with continuous states that keep the computation.

Small modules that predict the thinking state

The first attempts trained small external modules to produce the state thinking would have produced.

A 5M-parameter proposer and refiner produced 32 latent slots, trained only on the answer. On 200 problems it scored 90, against 94 for the untouched model with thinking off. Shuffling its slots between problems scored 99. It failed its own success criterion: it did not beat the baseline, and swapping slots did not remove a gain.

Transition predictors tried a different target: predict the full state after thinking, in a compressed space, and decode it through the frozen model. On a 40-problem evaluation they reached a mean squared error of about 0.39, well under the 0.87 of predicting the average state. They decoded 0 of 40 answers. Random noise of the same error size decoded 28 to 32 of 40.

Scalar error said the predictions were close. The answers said the direction of the error mattered more than its size. Error was also higher on positions holding digits (0.53 against 0.39 elsewhere), a hint, not proof, that the problem-specific calculation was what got lost.

Two more diagnostics narrowed it down. The predictor could memorize one known transition almost exactly, but not eight; its output layer was too narrow to fit the targets exactly, a rank limit that more training or more recurrent depth could not remove. And when we replaced the state with an average per answer label, the frozen model answered 200 of 200 if the correct label was supplied, and 105 of 200 with the label the predictor picked. For four-choice questions, the state did not need to reconstruct a reasoning trace. The predictor still had to pick the right answer, and that was where it failed.

A score that did not depend on the vectors

The next design trained the predictor jointly with a small adapter (LoRA) inside the language model. Scores went up: 108 of 200 against 94 for thinking off.

Then we ran the controls.

Latent input to the trained model Correct /200
Predicted latents 108
Latents shuffled between problems 108
Latent route removed 104

All 200 answer choices were identical with shuffled latents. The adapter, which could see the full prompt, had learned to answer around the latent path. An adapter trained with no latents at all scored 126. A later variant scored 286, 287 and 287 out of 500 as trained, with its latent steps removed, and with them shuffled.

This set a rule for everything after: removed and shuffled controls come before any reading of a depth result. From here on the backbone stayed frozen and only the bridge into it was trained.

More refinement steps, same answer

A 45.3M-parameter recurrent refiner on the frozen model re-read the problem at every step and refined a guess of the end-of-thinking state. It had no step counter, so it could run any number of steps.

On 200 problems it reached 123 at step 1 and 121 to 125 across steps 1 to 16. It stayed stable past the depth it was trained for, and extra steps added nothing. Training stopped at a plateau before its planned length.

Arm Correct /200
Thinking off 94
Refiner, average slot (no problem information) 75
Refiner, wrong problem fed in 101
Refiner, step 8 124
Adapter with no latents 126

The gain over thinking off was partly problem-specific (124 against 101 with the wrong problem) and partly a generic learned slot. Per-problem records were not saved, so the 23-problem gap has no confidence interval. Iteration contributed none of the gain.

One step through the frozen model

The cleanest test kept the 2B model fully frozen with no adapter. A shared 1.05M-parameter bridge took the model's last hidden state, turned it into an input vector, and ran it through the model again, k times, before the answer. Only the answer was supervised, plus a weak hint at the end state; nothing on the intermediate steps.

On 500 problems, skipping thinking entirely (k = 0) scored 157. Text thinking scored 500.

Comparison, 500 problems Gain 95% interval
One latent step vs none +8.0 pts +2.8 to +13.2
Two latent steps vs none +10.0 pts +4.8 to +15.2
Two latent steps vs one +2.0 pts −0.8 to +4.8

One step helped. The second step was not significant. Both arms plateaued, we stopped the sweep before its planned length, and the four-step arm never started.

Two caveats. Removed and shuffled controls were not run on these arms, so we have not shown the gain depends on what the steps carry. And a first launch of this sweep never supervised the answer letter at all; we caught it, threw those scores out, and added a check that the answer loss is actually nonzero.

What this fixed was the goal: computation in vectors, not a compressed copy of written reasoning. The reference score is the model forced to answer directly.

Exact values do not survive the latent path

The next design asked whether vectors could carry exact values at all. The precision ladder replaced written thinking with 8 vector steps, each adding one more bit of precision to every value of the end-of-thinking state.

A perfect ladder would have been enough. With true bits, the frozen model decoded 40 of 40 on one set and 171 of 200 on another, close to the 173 the compression itself allows. The question was whether a 5.1M-parameter predictor could produce those bits.

It could not. Per-step bit accuracy was between 0.51 and 0.68, and the bit loss sat at 0.666 against 0.693 for coin flips when training stopped at a plateau. Answer letter accuracy, read from a single forward pass rather than generated, ended at 0.250 and 0.210 on the two evaluation sets, which is chance for four options.

The answer loss did fall. It fell because the decoded state became well-formed enough for the model to commit to some letter, not because the letter was right. Our likely explanation: the predictor's input was two pooled vectors per problem, asked to specify up to about a million values.

The lesson we carried forward: exact values need an exact channel. In Cortex that became exact payloads stored in memory and actions that point at them.

How much of the thinking does the answer need

This last test needed no training: take the model's finished text thinking, keep only N evenly spaced positions of its internal state, drop the rest, and ask for the answer. If a handful of positions carried the answer, a compact latent target would be realistic.

On 200 problems, text thinking scored 200, no thinking state 72, and thinking off 65.

Positions kept (N) Correct
1 63/200
2 61/200
4 71/200
8 78/200
16 110/200
32 141/200
64 16/20
128 18/20
256 19/20
512 20/20
Answer accuracy against how much of the thinking state is kept Line chart. Keeping 1, 2, 4 or 8 evenly spaced positions of the thinking state gives 30 to 39 percent correct, close to the 32.5 percent with thinking off. 16 positions give 55 percent, 32 give 70.5 percent, and 64 to 512 positions give 80 to 100 percent on a separate 20 problem set. 0% 50% 100% thinking off 32.5% 1 2 4 8 16 32 64 128 256 512 positions of the thinking state kept 200 problems20 problems
Answer accuracy against the number of thinking positions kept. The points from 64 up come from a separate 20-problem set.

With 32 positions taken from a different problem, the score fell to 17 of 200. With thinking cut off before the answer number first appears, 32 positions scored 88. A few positions carry little. The answer needs 32 or more, out of a median of 189 thinking tokens. The rows from 64 up come from a separate 20-problem set, where 512 positions keep the whole thinking on 16 of the 20 problems.

That closed this direction for us. In these tests, a state compact enough to predict from two pooled vectors did not hold the answer, and a state that held the answer was too large for the predictors we had built. The next stretch moved the arithmetic out of the vectors and into a calculator, and asked a narrower question: can a small latent policy decide which operation to run next? It worked in range and collapsed past it, which is part 3.

How far to trust this

Every result here is a single run on synthetic four-choice word problems from a small set of templates. None of it is replicated across seeds, and none of it says anything about latent reasoning in general. The recurrent refiner, the bridge sweep and the precision ladder were also stopped early, at a plateau or by us.

Two gaps matter most. The one-step bridge gain (+8 points) has no removed or shuffled control, so we cannot yet say the steps carry problem-specific computation. And the transition predictors and the precision ladder both took two pooled vectors as input. A predictor that attends over the full problem was proposed and not run, so those failures may partly be an input bottleneck.

Next in Cortex research notes
Near perfect in range, zero past it: the long-horizon collapse