The confuser was the model's own results: selective memory
With only the facts each decision needs in view, Cortex solved 91.2% of 50-call problems instead of 14.8%. Two thirds of its wrong picks were its own computed results, and a small learned reader brought it to 85.5-89.5%.
When we hid everything except the facts the current decision needs, Cortex solved 91.2% of problems needing 50 calculator calls. With its full memory in view it solved 14.8%. The policy was the same; only what it could see changed.
Cortex is research at Saina, not a product. A frozen 2B-parameter open language model encodes the text. A small recurrent latent policy does the reasoning over a memory of facts and results. An exact calculator does the arithmetic. The policy is trained only on problems needing 1 to 4 calculator calls and tested on much longer ones.
Part 4 ended with two failure modes removed. The goal stack (pending goals kept outside the policy) took depth out of the policy, and per-sentence encoding took position out of the input. What was left was a slow decline with the number of sentences. On the policies used here, decision accuracy fell from 98.6% when the goal sentence was among the first three to 83.1% when it was sentence 40 or later.
Showing only the right facts
Our hypothesis was candidate interference: long problems put more sentences and computed results in memory, so every decision chooses among more candidates.
The test was an oracle: for each decision, replace memory with exactly the entries that decision needs, taken from the program's dependency structure. If the policy still failed, the problem was execution. If it recovered, the problem was choosing.
It recovered almost completely. Results are greedy success (no search), pooled over three policy checkpoints from two seeds.
| what the policy sees | 20 calls | 50 calls |
|---|---|---|
| full memory | 40.6% | 14.8% |
| only the needed entries (oracle) | 96.6% | 91.2% |
| as many random entries as the oracle shows | 0% | 0% |
With the oracle, 99.9% of decisions were right at every length. The decline with sentence index disappeared: 100% at the first sentences, 99.8% at sentence 40 and later. Later goals, larger problems and longer execution history did not hurt by themselves.
Random entries in the same number gave 0%, so this is not about less context. Adding back only 8 irrelevant sentences on top of the oracle cut 50-call success from 91% to 35%. Whatever replaces the oracle has to be very precise, and its recall has to stay near 100%.
The seed gap moved too. The weaker seed solved 1% of 50-call problems with full memory and 90% with the oracle. The stronger seed went from 42% to 93%. Most of the difference between seeds was in handling interference, not in executing.
Premature calls, where the policy calls the calculator before a needed source is computed, fell from 11.6% of the relevant decisions at 50 calls to 0. Premature completion was interference too.
What it confused
Next we labelled every wrong pick by the kind of entry chosen. We expected similar-looking facts to dominate: the same person with a different quantity, or the same relation about different people.
They barely occur: 0.5% of wrong picks each.
Computed results are 67% of wrong picks, 72% at 50 calls. These are the policy's own earlier outputs sitting in memory. Per visible competitor, a computed result that has not been used yet outranks the correct action 34.1 times per 1,000. A result already used on another branch does so 32.6 times per 1,000. An input sentence does so 2 to 5 times per 1,000, and a constant never.
Wrong picks usually share nothing visible with the goal. The goal is "Jade has as many marbles as Bea and Omar together", and the policy picks "18 (Amir has 18 marbles.)".
The second pattern was combine goals, the ones with two sources ("as many as A and B together"). They are 10% of decisions and 55% of wrong decisions. With full memory they are 63-71% right, against 96-97% for single-source goals.
Execution itself was fine: wrong operations and wrong operand order are 0.3% of errors.
The weaker seed confuses computed results 2.5 to 3 times as often as the stronger one. On input sentences the two are almost the same.
So this is not ordinary semantic retrieval. It is computational state (which result belongs to whom, and whether it is still pending) plus two-entity binding at combine goals.
Facts versus results
To separate the two memories we ran two more oracles. One showed only the needed input facts and every computed result. The other showed every input fact and only the needed results.
| at 50 calls | full | facts only | results only | both |
|---|---|---|---|---|
| greedy success | 14.8% | 18.0% | 51.5% | 91.2% |
| combine goals, decisions right (all lengths) | 66.6% | 83.6% | 77.2% | 99.4% |
| premature calls | 11.6% | 10.1% | 2.1% | 0% |
Execution memory mattered most. Restricting results gave 5 to 10 times the gain of restricting facts at 20 and 50 calls. The weaker seed went from 1% to 45% with only the right results and stayed at 1% with only the right facts.
But the gains were far from additive: facts alone added 3 points, results alone 37, both together 76. Combine goals needed both.
So a learned reader has to cover input facts and computed results in one query, and return a set, because a combine goal needs two sources.
Every quantity here is used exactly once, so "already used means irrelevant" is a perfect rule on this task and false in general. We did not give the reader a "used" flag.
Is breadth the same problem?
Before training a reader we checked a different kind of length: many short problems at once. If breadth works and depth fails, the issue is binding results inside one dependency graph, not the number of goals.
A goal-stack policy with a request queue, trained on 1 to 4 concatenated problems, was run on 16, 20 and 32 (at 32, about 80 operations).
- Selecting the requested problem was 100% right at every count and every position in the text.
- In the request-swap test (same input, another problem requested), it picked the requested problem 100% of the time.
- Selection among the problems cost at most 0.3 points of per-problem accuracy, and switching between them at most 0.7.
Episode success did fall, to 54% and 28% at 32 problems for the two seeds. That is per-problem accuracy compounding: 97-99% and 95-97.5% for each problem solved alone. Of 500 composites across both seeds, exactly one failed where every one of its problems was solved alone.
The caveat: problem membership is given here. Each memory entry carries a flag saying whether it belongs to the current problem. That flag is the kind of state the interference analysis found missing inside a single long problem. Neither seed had finished training either.
So breadth works with membership given. The open problem is binding computed results to the right goals inside one long graph.
A learned reader
The reader is a 1.7M-parameter scorer. For each memory entry it takes the current goal and the entry's frozen sentence state, plus the goal stack's flags, and decides whether the entry belongs in the working context. It sees no values, no problem membership and no "used" flag, and nothing from the policy. It returns a set, capped at 6 entries per decision. Like the policy, it was trained only on problems needing 1 to 4 calls.
At 50 calls there are 86 candidates per decision on average, with 1 to 3 of them needed.
Retrieval was near exact:
- Every required entry retrieved: 99.87% of decisions at 20 calls.
- Precision 99.97% at 20 calls and 99.86% at 50.
- On states reached after the policy's own mistakes, recall was still 100%. What fails there is the earlier wrong value, which retrieval cannot undo.
We put the reader in front of the unchanged policies.
| policy | full memory, 20 / 50 calls | with the reader | oracle |
|---|---|---|---|
| stronger seed | 56.7 / 41.5 | 96.3 / 89.5 | 98.9 / 93.0 |
| weaker seed | 29.7 / 1.0 | 93.3 / 85.5 | 95.7 / 90.0 |
| pooled, three checkpoints | 40.6 / 14.8 | 94.2 / 86.8 | 96.6 / 91.2 |
The seed gap at 50 calls went from 40.5 points to 4.0. Premature calls went from 11.6% to 0.1%. The decline with sentence index was gone, 100.0% to 99.8%, as with the oracle.
At 50 calls, 79 of 600 episodes failed. 27 had a retrieval miss, all at combine goals, which accounts for about the 4.4-point gap to the oracle. The other 52 had exact retrieval at every step and failed on execution errors the policy also makes under the oracle.
A second reader seed replicated it: 94.5% at 20 calls and 88.0% at 50, pooled over the same three policy checkpoints.
On this task, the seed gap was interference handling, not execution, and selective memory access became a fixed part of the architecture.
The first frozen milestone
At this point we froze the stack as Cortex's first milestone: the frozen encoder with per-sentence encoding, the goal-stack policy, the learned reader and the calculator, trained on 1 to 4 operations. Later parts are measured against it.
What it shows on this task: long-horizon execution, selective memory, and goal management across 32 problems with membership given.
What it does not show: new wording, a new domain, or recovery. In earlier experiments about 78% of long failures were an off-course call followed by a confident early answer, and nothing here fixes that.
These are existence results on one synthetic arithmetic domain with fixed sentence templates and unique names per quantity, where matching a source name to a subject name is easy. The next part, It learned the template, not the relationships, tests what happens when the wording changes.
What the reader does not fix
The reader takes away interference. It does not take away error. About 9% of 50-call problems fail with perfect retrieval, because the policy has no way to recover from a wrong value. Every remaining retrieval miss sits at combine goals, about 1.3% per combine decision.
The reader also needs labels from the program's dependency structure to train, and here those come from short problems. Breadth was shown with problem membership supplied as a flag. Joining the reader with the multi-problem goal queue is not tested.
These results are two policy seeds and two reader seeds on one domain. The question we carried into the next domain is whether any learned weight is about reasoning, or only about arithmetic dependency graphs.