Research · November 17, 2026

It learned the template, not the relationships

Reworded as paraphrases, the same 50-call problems Cortex solved 87.5% of the time dropped to 0%. It had learned the template, not the relationships, and reading is now the limit.

Mendrika Ramarlina · November 17, 2026 · 10 min read

The first frozen milestone of Cortex solved 87.5% of problems needing 50 calculator calls. We re-rendered the same problems as paraphrases, with the same numbers and the same dependency graph, and it solved 0%.

Cortex is a research system at Saina, not a product. A frozen 2B-parameter open language model encodes each sentence of a problem. A small learned reader picks which facts and results from memory are relevant to the current goal. A small recurrent policy, the executor, decides what to do next with a goal stack, and an exact calculator does the arithmetic. Part 5 ended with that milestone: long chains, selective memory and goal management all working in one fixed family of sentence templates.

Same problems, new wording

To test whether the milestone reads relationships or templates, we took 950 problems needing 4, 8, 20 and 50 calculator calls and rendered each one seven ways: the original template, the same sentences shuffled, paraphrases, an inverted perspective (the source quantity as grammatical subject), repeated names (one person with two quantities), pronouns, and another surface domain. Nothing was retrained. Oracle retrieval below replaces the reader with the correct entries, so only the executor is tested.

Greedy success (no search), both executor seeds pooled, problems needing 4 / 8 / 20 / 50 calls:

Rendering Learned reader Oracle retrieval
Original 99.2 / 99.8 / 95.4 / 87.5 100 / 99.8 / 97.4 / 91.5
Shuffled identical identical
Paraphrase 1.4 / 0 / 0 / 0 17.8 / 3.8 / 0.6 / 0.25
Inverted perspective 0 / 0 / 0 / 0 0 / 0 / 0 / 0
Repeated names 0 / 0 / 0.2 / 0 15.8 / 12.2 / 8.2 / 6.75
Pronouns 0 / 0 / 0.2 / 0 0.2 / 0.2 / 0.2 / 0
Other domain 0 / 0 / 0 / 0 1.6 / 0.8 / 0.2 / 0
The same problem in two wordings Two wordings of one problem share the same numbers and the same dependency graph. With the original wording the system solves 87.5 percent of problems needing 50 calls. With the paraphrase it solves 0 percent. original wording paraphrase same numbers,same graph Solved at 50 calls 87.5% 0%
Same numbers and same dependency graph, different wording. Learned reader, problems needing 50 calls, nothing retrained.

Shuffling changes nothing, by construction: each sentence is encoded alone and there is no position input. Every other rendering loses 85 to 97 points at 20 and 50 calls.

Both learned parts break. The reader does not pick wrong entries. It stops recognising relevant ones, retrieving 0 to 0.5 entries per decision where 1.1 are needed, and even a far lower threshold lifts its paraphrase recall only to 55.7%. Its exact recall at 20 calls falls from 99.9% to 33.6% on paraphrases and to 0% on the inverted perspective and the other domain.

The executor, given exactly the right entries, still fails. Its per-decision accuracy at 50 calls is 96.5% on repeated names, 78.6% on paraphrases, 62.4% on the other domain, 44.9% on pronouns and 28.8% on the inverted perspective. Even 96.5% per decision compounds to 6.75% success over 50 calls.

The inverted perspective, where the relation is stated correctly and only the grammatical subject changes, is the cleanest evidence: both parts behave as if a sentence defines its grammatical subject.

An adapter in front of a frozen core

Next we tested whether the recurrent mechanism itself is reusable. We put a small residual adapter between the frozen language model and the frozen reader and executor, and trained only the adapter on three new wording families plus repeated names. Arm A trained the adapter alone (6.76M parameters). Arm B also trained the input readouts of the reader and executor (another 4.07M). One seed per arm.

Greedy success with the learned reader, problems needing 20 / 50 calls (one executor seed, so the milestone row differs slightly from the pooled table above):

Wording Milestone Arm A Arm B
Original template 96.8 / 89.5 92.0 / 84.5 96.0 / 86.5
The three trained families 0.4 / 0 85.6 / 73.0 91.2 / 80.0
Repeated names 0 / 0 37.2 / 12.5 54.4 / 51.0
An unseen near-paraphrase family 0 / 0 0.8 / 0 0 / 0
Unseen source-first structure 0 / 0 0 / 0 0 / 0
Unseen surface domain 0 / 0 0 / 0 0 / 0

With oracle retrieval, arm A on the trained families reaches 96.8% at 20 calls and 91.5% at 50. One trained family puts the referenced quantity first ("Mia has some apples, and Joel has 2 times that many"), and on it the milestone's decisions go from 12% to 99.3%.

So the frozen core runs on adapted representations, at least for wording the adapter was trained on. That part of the mechanism is reusable.

What does not carry over, as far as this run can tell, is the mapping into that representation. On wordings the adapter never saw, the best arm gets 22 to 37% at 4 calls, 0 to 1% at 20 and 0% at 50. Per-decision accuracy on near-paraphrases rises from 30 to 50% to 75 to 85%, not enough for long chains. The unseen source-first structure is read in the wrong direction, and the reader finds nothing in the unseen domain. Training the input readouts as well (arm B) does not help on unseen wording.

Repeated names are the one place the readouts had to move: with oracle retrieval the executor is at 99.9% of decisions in both arms, so telling two quantities of one person apart is a retrieval problem.

There was a cost on the original template. Arm A lost about 5 points with the reader at 20 and 50 calls, and 19 points at 50 calls with full memory (the executor choosing from everything stored, with no reader). The check we watched during training used 50 problems needing 4 and 20 calls and stayed at 100%, so it missed the loss.

Role reversal ("Mia exceeds Joel by 3") stayed at 0. The adapter reads such a sentence as the right equation, but it still treats the subject as the quantity being defined. No sentence-isolated encoder can pick the defined quantity; that needs the rest of the problem.

Retrieval that follows a chain one hop at a time

A parallel line of work cut problems needing 20 calls into subproblems of at most 4 operations, using the correct decomposition as an oracle. A learned selector picked the sentences each subproblem needs, and three frozen executors trained only on short problems solved them.

A one-shot selector (6.3M parameters) worked on short problems and on composites longer than anything in training. On real 20-call chains it failed. Complete-set recall, meaning every required sentence retrieved, was 47 to 62% across thresholds, and end-to-end success was 0%.

The misses were concentrated in one kind of fact: the earlier result that the current subgoal builds on, missed 303 times out of 586. Two variants of the selector, trained on more known results and on longer connected chains, stayed at 53 to 58%.

The fix changed the interface, not the model. The same selector got one extra input bit, "already retrieved", and ran one hop at a time for up to 6 hops, predicting only the immediate dependencies of what it had just retrieved. Complete-set recall at 20 calls went to 99.0%. Evaluated end to end on 100 problems needing 20 calls, the pipeline solved 83 to 85% across the three executors, against 95 to 97% with oracle retrieval on the same problems.

Merging the planner, the reader and the executors

That pipeline still used oracle subgoals. The goal stack from part 4 already discovers the dependency graph as it runs: every calculator call names the sentences it uses. So we used it as the planner. The goal stack runs, its calls give the graph, a fixed rule cuts the graph into subproblems of at most 4 operations, the incremental reader retrieves facts for each, and the frozen text executors solve them. All components were frozen and trained separately.

On the same 100 problems needing 20 calls:

Pipeline Success
Goal stack as planner, incremental reader, text executors 82–83%
Oracle subgoals, incremental reader, text executors 83–85%
Oracle subgoals, oracle retrieval, text executors 95–97%
Goal stack alone, incremental reader in front 96%

The goal stack's decomposition matched the oracle's on 96% of these problems, and replacing the oracle subgoals cost nothing measurable: the planner is not the limit.

What limits the merged pipeline is per-subproblem retrieval and the handoff to the executor. The goal stack on its own does better on the same problems (96%), because its retrieval is conditioned on the stack state and it carries real results between operations.

Real wording: GSM8K as expression trees

The goal stack expects one defining sentence per step. Grade-school word problems are not written that way, so we compiled each worked solution into a top-down expression tree, where each equation step is a goal. Of 30,000 problems, 27,394 compiled (91.3%), and an exact calculator evaluating the trees reproduced every gold answer with 0 mismatches (the compiler already required the last step to match, so this checks the tree structure). 97% have depth 4 or less, 40% use a constant not in the question (the top 20 constants cover 89.6%), and 13.4% reuse an intermediate result.

A 9.9M-parameter node policy on frozen language-model features decides the operation and the two operands at each node, trained on trees of depth 4 or less. On 452 GSM8K problems it answered 56.9% correctly, and 25.7% on a harder set of 1,282 GSM8K problems from our long-horizon benchmark.

Oracle controls located the error: the correct operation lifts 56.9% to 60.6%, the correct operands to 76.8%. All decisions correct gives 98.7%, so the tree, stack and calculator are sound. Showing only the true number mentions gives 55.1%, no gain, so retrieval is not the lever. On the 269 problems where an earlier attempt, which adapted the goal-stack policy directly, has a stored outcome, the node policy got 59.9% against 24.5%.

On the 452 problems, constants were the weakest operand, and the errors followed cue words: "half" against "third", "twice", "dozen", number words, unit conversions. The policy saw only sentence summaries, so it could not read the cue. Adding the frozen model's state for every question word lifted answers to 62.2% on the 452 problems and 27.8% on the harder set, and operand accuracy on constants from 43 to 59% and from 46 to 57%. One honesty note: 53 of the 452 problems closely resemble problems the policy learned from (the same numbers and the same sequence of operations), are answered about 85% of the time and add 3 to 4 points to the figures on the 452, and without them word-level reading still gains 6.1 points (53.1% to 59.2%).

Depth 5 and beyond did not move: on the harder set those problems stay at or below 9%. The policy never trained on them (they were 3% of the compiled trees), and on that set earlier results used as operands are right only 30% of the time. That is a data and depth question, not a reading one.

Apart from GSM8K depth, most failures in this part sat in reading the input, not in the recurrent core. Part 7 takes the same question to a second domain, where the inputs are tool calls instead of arithmetic sentences.

Where reading stands

Unseen wording is unsolved. An adapter trained on three families does not transfer to new families or a new domain. Whether broader wording training would transfer, or whether the encoder needs context across sentences, is untested. Pronouns and role reversal need that context, and nothing here addresses them.

In the merged pipeline the gap is per-subproblem retrieval and the handoff: 82–83% against 95–97% with oracle retrieval. On GSM8K, depth 5 and beyond is unsolved. Earlier results used as operands are right 30 to 60% of the time, and word-level reading did not help them. Training on longer trees breaks "train short, test long" and has not been run.

On seeds: the adapter results are one per arm, and differences of a few points between arms are within seed noise seen elsewhere. The incremental reader and the merged pipeline are one seed each, and the GSM8K node policy is single seed for the word-level version.

Next in Cortex research notes
Beyond arithmetic: a tool-use world and a general action interface