Shlexball training: Pristine and the supervision repair

Since the 12 October research note, Shlexball has gained a stronger experimental model that answers without semantic repair by the app. Its preserved result is 580/733 executed development tests and 61/89 on a separate confirmation set. The next targeted training round did not improve that model. We then traced weaknesses in its training supervision, repaired the accepted data and strengthened the tests used to validate it. No model has yet been trained on this latest repair.

This note covers the completed work through 14 October: the Pristine restart, a revised evaluation contract, the successful first continuation, an app-assistance conflict, the unsuccessful second continuation and the supervision repair. The recommended next experiment is controlled supervised fine-tuning on the repaired data. Reinforcement learning remains conditional on evidence that its rewards can distinguish useful answers.

Pristine began from pretrained Qwen2.5-Coder-1.5B-Instruct, with full-weight training, a consistent document prompt and no frozen v6 teacher. Starting from stock Qwen retained its pretrained knowledge. The cleaned corpus contained 17,729 unique examples, with oversampling producing 18,191 exposures per epoch. Three epochs completed 3,411 optimizer steps. Initialization, data and prompt changed together, so this run did not isolate whether stock initialization was better than continuing v6.

Early fresh-validation results rose from 7/76 at step 128 to 25/76 at step 1,137 and 27/76 at step 3,411. The final model passed 435/703 on the historical direct-output metric, below v6 at 506/703 and v6.1 at 573/703. Those older models passed only 2/76 and 9/76 on the fresh cases. The mixed result prompted an audit of both genuine planning failures and disagreements between the old test expectations and the new document contract. We retained the original scores.

That audit mattered especially for file types. Ordinary document topics should not implicitly search environment files; unrestricted requests for actual stored credentials should include them. Some historical expectations disagreed with that distinction. We created the separately versioned Findbench 2.0 development board: 657 independently reviewed historical cases plus the 76 fresh cases, for 733 total. Review made 211 type-policy corrections and excluded 46 historical cases: 26 semantic quarantines and 20 without usable independent review. Missing review was not evidence that a case was wrong.

A pass requires the model's first answer to execute through the frozen native broker and return exactly the expected paths, including applicable order and limits, or the exact expected refusal. Execution must complete without a safety violation. The scorer supplies no semantic correction. The 76-case subset is included in the 733; it must not be added again. This board has informed training design and is not an untouched generalization test. Its scores are not directly comparable to the earlier 703-case app-assisted results.

Model comparison on Findbench 2.0. At the operator’s request, we also ran the latest text models in Google’s Flash and Flash-Lite families as of 14 October: Gemini 3.8 Flash and Gemini 3.5 Flash-Lite. Both explicitly used medium thinking and the same 30 worked examples. Every model below is scored on the same frozen 733 cases, without semantic app assistance.

Pristine repair, v6 and v6.1 each have 1.5B parameters. Parameter counts for Gemini 3.8 Flash and Gemini 3.5 Flash-Lite are undisclosed.

These are fresh Gemini runs on Findbench 2.0. The archived 312-case comparison used a different board and app processing; those scores cannot be carried over. Both models’ exact returned IDs matched the requested IDs. The worked examples, responses and per-case records remain private.

The first continuation tested a concrete data change. Both arms started from Pristine step 3,411 and trained for 400 more steps with the same prompt, seed and optimizer settings. The control continued the clean broad mixture. The repair arm used 75% broad replay and 25% from a corrected 499-example pool. A separate 89-case confirmation set supplied an additional check after candidate selection.

The repair gained 71 development passes and lost 20 against the matched control, a net gain of 51. On confirmation it gained 26 and lost two, a net gain of 24. It passed 43/71 supported searches and 18/18 genuine refusals there. All five models in the wider confirmation comparison recorded zero unsafe results. This was a useful measured improvement, with remaining regressions and only one training seed. It did not establish universal superiority or release readiness.

The checkpoint was preserved as Pristine_Repair_Unaided, together with its prompt, tokenizer, broker and scored evidence. Its 580/733 development result is 79.1%; its separate 61/89 confirmation result is 68.5%. The fresh subset remained 29/76, including 0/11 PDF concept-composition cases. Much of the aggregate gain came from retained search abilities. Once examined, the 89 confirmation cases became regression evidence for subsequent work; they could no longer serve as fresh confirmation.

We also checked the application boundary. Existing app assistance changed the repaired model's results from 580/733 to 585/733, but lowered confirmation from 61/89 to 60/89. A legacy recovery path replaced a correct unsupported-format refusal with an incorrect search. The execution was safe, but the answer was wrong. This demonstrated a specific recovery conflict, not a failure of every native search plan.

A private local preview now sends the model's first answer unchanged through normal native validation and execution. It bypasses semantic rewriting and helper turns while retaining readers, permissions, confinement, write denial and credential redaction. All 822 saved answers passed through the app routing unchanged, and fresh execution reproduced 580/733 and 61/89 with identical outcomes. App checks passed 287 tests, with seven ignored. This correction remains in the private preview; production-source integration and release validation are still outstanding.

For context, replaying the older assisted recipes on the same 733-case contract produced 481/733 for v6 and 466/733 for v6.1, against 580/733 for the unaided repair. These compare complete model, prompt and app recipes, not weights alone. The Findbench 2.0 definition and aggregate results were published separately on 13 October. The later training work below has not raised that published baseline.

The second continuation added 534 focused examples: 448 searches and 86 refusals covering compound requests, Boolean logic, conversational corrections, command structure and composed constraints. We froze a new, separate 193-case confirmation set before training. Both arms started from the preserved repair checkpoint. The control drew 12 broad and four prior-repair examples per 16-row batch; the focused arm drew nine broad, three prior-repair and four new examples. Thus “focused” means a changed training mixture, with one quarter of each batch devoted to the new pool.

Both arms used 400 optimizer steps, learning rate 5e-6, 25 warmup steps, cosine decay, fresh AdamW state and the same seed, prompt and native broker. Training used BF16 computation with FP32 optimizer/master weights, microbatches of four, weight decay 0.1 and gradient clipping at 1. There was no teacher or KL preservation term. We saved steps 128 and 400, and evaluated each model's own first answer without app assistance.

No candidate passed the frozen advance gates. Even the apparent tie at focused step 128 traded 17 newly passing development cases for 17 lost passes. The predeclared primary comparison, focused 400 versus control 400, gained 20 and lost 19 on development, then gained none and lost six on the prior confirmation set. All four snapshots reached 1/11 on PDF concept composition, but that narrow gain did not offset their regressions. Zero unsafe executions were observed. We retained the baseline and left the new 193-case confirmation unopened.

The second round ran on a RunPod Secure Cloud L40S. Its observed account debit, including storage, was about $0.96. All four full checkpoints were recovered and verified by size and SHA-256 before the pod was deleted at 10:08 p.m. Vancouver time on 13 October. A live check on 14 October found zero pods. The later diagnostic and supervision-repair work ran locally. Earlier, the initial Pristine launch had also required replacing an unreachable Ada pod before any data upload; the successful L40S run's checkpoints were recovered before deletion.

The next question was whether the focused model had learned the examples at all. A fixed diagnostic selected 200 training cases by hash order, with 40 from each family, before examining model outputs. The baseline passed 69/200, control 128 passed 72/200, control 400 passed 70/200, focused 128 passed 104/200 and focused 400 passed 184/200. The final focused model passed 152/168 searches and 32/32 refusals, gaining 115 cases and losing none against the baseline on this sample. By step 400 every focused example had been encountered two or three times.

Those are training-data results, with equal family quotas rather than population weighting. They demonstrate learning of the sampled supervision, not unseen accuracy. Their coexistence with development regressions does not establish a hard capacity plateau, pure memorization or a specific optimizer failure.

The supervision audit found a more immediate problem. All 534 original reference answers passed their original fixtures. Yet removing individual constraints produced 1,379 valid weakened plans, of which 316 still passed. In total, 212/448 search fixtures missed at least one weakening; all 11 folder-exclusion cases still passed when the exclusion was removed. This measures fixture sensitivity, not the percentage of wrong targets: some removed clauses can be redundant.

Inspection also produced five concrete semantic counterexamples. Their targets dropped essential topic qualifiers, made a date unnecessarily contiguous, omitted closure intent or treated a standards mention as certification. All five passed the original fixtures and failed extended ones. Separately, an audit of all 19,224 assembled training rows found no conflicting targets for identical request/context pairs, including normalized comparisons. That check does not rule out contradictions between differently worded requests.

The loss calculation supplied another hypothesis to test. It averaged over answer tokens, giving longer answers more objective weight. Refusals represented 7.84% of rows in the focused run but only 1.75% of its objective. The new pool was 25% of rows and 26.87% of the objective. These are mechanical objective shares, not measured gradient effects or proof of why accuracy regressed. The final focused training diagnostic already passed all 32 refusal cases.

The completed repair preserved every original request verbatim and audited all 534 targets. A fresh automated target review flagged 106 for investigation; combined with fixture weaknesses, that selected 266 rows for repair. Flags are review judgments, not an error-rate estimate. Authoring and reviews used stateless, tool-free Claude Opus 5.5 calls at medium effort through the subscription CLI. Reauthored cases received separate relevance and target reviews: the relevance reviewer could not see the answer plan, and the target reviewer could not see the fixture documents. Unchanged cases retained their original relevance approvals and received the new target audit and execution checks.

Each retained reference had to execute correctly, and every tested valid single-constraint removal had to fail. We added plausible alternative positives and negatives that distinguish individual requirements, including matching material inside excluded folders. After one bounded follow-up, the accepted set contains 518 rows and 16 quarantines. Five quarantined rows lacked complete follow-up reviews after malformed output; other failures involved semantics, execution or insufficiently distinguishing fixtures. Quarantine does not mean all 16 requests were intrinsically wrong.

Reviewer agreement still missed an error: both automated reviewers accepted a repair that conflated testing against a standard with certification. Subsequent assistant inspection caught it, and added positive and negative probes forced a further correction. The reviewers are automated judgments, not human ground truth. Passing the strengthened tests establishes these specific checks, not exhaustive semantic correctness.

The broad 18,191-row pool and prior 499-row pool were copied byte for byte; their semantics were not newly audited in this repair. All 19,208 assembled rows passed production token limits and exact tokenizer round trips. Five regression tests for the gate passed. The original datasets, frozen benchmark, prompt, broker and preserved model remain unchanged. The repair is a separately versioned private data package; it has produced no new model accuracy score.

Recommended next steps. We should test the corrected supervision before spending another run on a larger model or assuming reinforcement learning will solve the remaining errors.

1. Run a matched supervised fine-tuning experiment from the preserved baseline, using only the 518 accepted focused rows. Keep the 9 broad / 3 prior / 4 focused mixture, learning rate and 400-step ceiling fixed. Compare the existing token-mean objective with an objective that first averages each answer's token losses, then averages examples. Use identical draws and seeds. This isolates weighting while giving both arms the repaired data; it does not isolate the effect of the data repair itself.

2. Rehearse the objective and gradient accumulation locally before provisioning. Freeze a new protocol and save early snapshots at 64, 128 and 256 as well as 400, within a verified recovery budget. Judge every snapshot by executed tests and retained individual cases. Keep the existing gates: no lost baseline passes on either development board, zero unsafe results, at least ten additional passes on the 733, five on the prior 89, one on the fresh subset and at least one PDF concept pass. A second seed would strengthen confidence if the budget permits.

3. Select and freeze a qualifying candidate before evaluating the new 193-case confirmation set once against the preserved baseline. Keep that set out of training and tuning. Recover full weights, verify their hashes and delete the GPU promptly; remaining local scoring does not require an allocated GPU. Integrate and validate the preview's unaided routing separately before treating a production app as equivalent to these model scores.

4. Before deciding on Group Relative Policy Optimization (GRPO), sample eight direct plans for each of a fixed, balanced set of 64 training-only probes. Report normal first-answer accuracy separately from any-of-eight success, mixed correct/incorrect groups, all-wrong groups and unsafe outputs. If useful correct plans already appear among alternatives, a small GRPO pilot has a concrete reward distinction to exploit. If most groups are all wrong, improve supervised coverage first.

5. For any GRPO pilot, reward the candidate's own single answer by exact native execution across varied, independently reviewed fixtures. Retain the weakened-plan checks as tests of the reward itself. The earlier five October 10–11 GRPO jobs passed no complete promotion gate and used an assisted runtime with a teacher-helper mismatch between training and evaluation. That history does not establish that GRPO will fail on the present model, but the old reward pipeline should not be reused unchanged.

DeepSeekMath, sections 4.1–4.2, describes GRPO's group-relative baseline without a separate critic. With identical binary rewards, a group provides no relative correctness advantage, although a KL term can still contribute gradients. Checking reward variation first is our proposed application of that method; the paper does not establish gains for Shlexball. The paper has also been added to the repository's reading notes.

We have not established a model-capacity plateau. We have established a failed continuation recipe, strong learning on its training examples and concrete supervision defects. Corrected supervised comparisons and trustworthy execution rewards are the next tests. A larger base model becomes a better-supported next step if those controlled experiments still stall on independent executed evaluations.