Removing depth and position as failure modes
Encoding each sentence on its own took decision accuracy on problems needing 20 calculator calls from 29.5% to 94.5%, with the same policy trained only on 1 to 4 calls. Getting there meant ruling out depth first.
Encoding each sentence of a problem on its own took decision accuracy on problems needing 20 calculator calls from 29.5% to 94.5%. The policy, mechanism and parameter count stayed the same, and it was trained only on problems needing 1 to 4 calls.
It came after a run of experiments that each removed one suspect. Most were negative, and they pointed at position.
A short reminder of the system. Cortex is a research system at Saina, not a product. A frozen 2B-parameter open language model encodes the text. A small recurrent latent policy does the reasoning over a memory of facts and results. A calculator does the exact arithmetic. "Greedy" below means success with no search. "Oracle" means one component is replaced by the correct answer, to isolate the rest.
Part 3 ended with the policy near perfect on problems needing 1 to 4 calls and at 0% on problems needing 8 or 20. Probes showed the needed information was in the state. The decision built from it was what failed.
Not the growing table
Our first suspect was attention over the whole results table. On a 20-call problem the table holds far more results than anything seen in training, so we built a goal-relative view. Each memory slot's state depends only on the results it was derived from, and the answer is scored against a goal token.
In range, every variant learned problems needing 1 to 4 calls at 99–100%. Past range, nothing moved. Problems needing 8 calls stayed at 0–2% solved, 20 calls at 0–1%. Per-step accuracy along the correct solution stayed at about half at 8 calls and about a sixth at 20, the same as the baseline policy from part 3. Locality alone, the goal token alone and both together were indistinguishable.
These runs were stopped at about 40% of training and monitored on 100 problems per level, but step accuracy showed no trend over training. Attention over the growing table is not what limits these policies on these problems.
Two failures stacked
So we separated two things the long problems mix together: irrelevant history and genuine depth. This was training-free. We took short problems the policy already solves, stopped after 2 calls, and inserted 2 to 16 unrelated calculations before asking for the next action.
The baseline policy went from 100% correct to 0% once 8 unrelated calls were added, answering prematurely in 96% of states. The goal-relative policy held 90.7% at 8 inserted calls and 73.3% at 16. Locality largely removes sensitivity to irrelevant history.
With relevant history instead (states on long problems after k completed calls that all matter), the local policies collapse too. After 4 relevant calls they are at 22–25%, after 10 at 0%, and premature answers rise with relevant depth.
Two failures were stacked. Locality removed the first one. The second, carrying a decision past about 4 dependent steps, is why the goal-relative runs did not move long-problem success.
Both halves fail
The policy makes two kinds of decisions: whether to stop and answer, and which call to make. We gave each one the oracle in turn, evaluated on 250 problems per level.
With the stop decision given, the learned call choice burned the whole budget. Only 9% of episodes ever answered, with about 11 wasted calls out of about 15. With the calls given, the learned stop still answered prematurely in 95% of episodes. For the baseline policy, neither oracle lifted problems needing 8 or more calls above 2%. The goal-relative policy with oracle actions reached 22.4% at 8 calls, still far from usable.
Knowing which calls to make isn't the same as knowing when the goal is resolved, and the policy had neither past its range.
Bounded subproblems run at any depth
Next we asked whether the short-range rules could be reused on a pre-cut problem. An oracle decomposed each 20-call problem into subproblems of at most 4 operations, each rewritten as a self-contained text and run as a fresh episode, with results carried forward.
The frozen short-range policies, including their own learned stop decision, solved 95% (goal-relative) and 99% (baseline) of 20-call problems this way. The same policies solved 0% of the same problems presented whole. Passing the calculator's actual results between subproblems instead of oracle values gave the same numbers.
Execution inside a bounded goal is reusable at any depth. The open question became what part of the rewrite does the work.
Swapping only the question sentence of the original text to name the subgoal gave 0%. The first call was unchanged by the swap in 100% of problems: the policy works through the text in its habitual order and does not read which quantity is asked. Keeping only the relevant facts in their original order gave 73–75%. The full rewrite gave 95–99%.
The goal is read in range only
Our hypothesis was that the policy had never needed to read the goal, since in training the asked quantity was always the last one. So we trained on counterfactual questions: the same facts, a different quantity asked.
In range it worked. On counterfactual questions the policies made the right first call 99.7–99.9% of the time and solved 99.0–99.5%. The baseline made the right first call for each question 58.5% of the time. On long text it did not. At 20 calls the first call was still unchanged by the goal swap in 96–100% of problems, and every subgoal condition stayed at 0%.
A position test made the reason visible. With the relevant facts placed first and unrelated problems after, success fell gradually, to 62–83% with 8 unrelated problems. With the relevant facts placed last, success was 0–4% even with 2 unrelated problems in front. The goal decides what is read inside the facts the policy attends to. Where the work starts is still set by text order.
A goal stack removes depth, and exposes position
To take depth out of the policy entirely, we gave it an explicit goal stack. Its actions became push a subgoal, make a call or answer the current goal, so every decision is about one bounded goal no matter how deep the stack is.
Decisions about goals whose sentence starts in the first 50 tokens of the question were 97–100% correct at every stack depth, including 16 or more goals stacked. Within that band, stack depth had no effect. What was left showed up as a function of where the sentence sits in the question: 98.3% in the first 50 tokens, 49.6% at 50–100, 11.6% at 100–150, 0.9% past 300.
Greedy success rose at 6 calls, from 4.4% for the baseline to 51.4%, but was 1.4% at 8 and 0% from 12 on. Longer training did not widen the range; the long-problem monitors were flat early. That became a working rule: when long-horizon accuracy is flat, change the representation, not the training length.
In parallel we tested a dependency radius, how many hops of derivation each slot can see. With radius 4 or 8, nearly all late failures were premature answers, which fits a shortcut learned from short problems: a result 4 steps deep is always the answer. Radius 1 lifted 6-call success to 41–51% against 13.6% for radius 4, and the number of layers did not matter. With one seed, it still failed by 8 to 10 real calculations, so visible depth is one cause of premature answers, not the whole one.
Encoding each sentence on its own
The goal stack read every sentence through one encoding of the whole question. So we encoded each sentence alone, and a sentence's representation no longer depends on where it sits. Same policy, same parameters, same training range of 1 to 4 calls.
Decision accuracy along the correct path, with oracle goals, two seeds averaged:
| Calls in the problem | Whole question | Per sentence | Difference [95% CI] |
|---|---|---|---|
| 8 | 78.4% | 98.3% | +19.9 [18.9, 20.9] |
| 12 | 51.6% | 97.0% | +45.4 [44.5, 46.3] |
| 16 | 37.9% | 95.6% | +57.8 [57.1, 58.5] |
| 20 | 29.5% | 94.5% | +65.0 [64.5, 65.6] |
| 50 | 11.2% | 88.2% | +77.0 [75.8, 78.2] |
By token offset of the goal sentence, the whole-question encoding falls from 98.2% to 0.9%. The per-sentence encoding goes from 97.7% to 85.7%.
Greedy success at 20 calls went from 0% to 56.8% for one seed and 29.2% for the other. At 50 calls, from 0% to 41.5% and 1.0%. Per-decision accuracy differed between the seeds by only 2–11 points, but a 50-call problem needs about 100 decisions, and the gap compounds.
The remaining decline has no positional input behind it. It tracks the number of sentences. Our read, not yet tested here, is that more sentences means more candidates competing in each choice. That is the subject of part 5.
A rule instead of a learned rewriter
One gap from the subgoal tests remained: relevant facts in original order gave 73–81% at 20 calls across the executors, the full rewrite 95–99%. Before training anything, we compared the two texts across all 897 subproblems.
Selection, wording and how carried results were stated were identical. The texts matched exactly in 816 of 897. The other 81 differed only in where plain given sentences ("Ivan has 54 shells.") sat. On those 81, the original order was solved 25.9–53.1% of the time, the rewrite 98.8–100%.
So the rewrite's only job here was to place each given right before the sentence that first uses it. A deterministic rule that does exactly that, using only the text, matched the rewrite in 888 of 897 subproblems and reached 98–100% at 20 calls with the frozen executors. No learned component was needed on these problems. We now treat that as a rule: don't train a component that a rule can replace.
What this does not solve
- Depth and position are removed for this policy on these synthetic problems. Long problems are still not solved reliably: 56.8% and 29.2% at 20 calls across two seeds, and seed variance is the largest open issue.
- The subgoal results use an oracle decomposition. A learned decomposer that produces subproblems like these was not tested here.
- Counterfactual training did not make the goal causal on long text. The goal still does not select where reading starts.
- Radius 1 cut premature answers but did not extrapolate. We stopped that line rather than resolving it.
- The ordering rule depends on sentence templates and capitalised names. It is not tested on other renderings or free text.
- Several arms ran one seed, and the goal-relative arms are early checkpoints, not final weights.