Lessons from Shlexball training since v6

The latest experiment improved some capabilities but did not produce a better overall Shlexball model. The final checkpoint passed 654/703 executed tests, compared with 666/703 for v6 and 647/703 at checkpoint 666. PPT improved from 28/57 at the intermediate checkpoint to 31/57, but remained below v6's 37/57. Other tests finished at 623/646 against v6's 629/646. These comparisons used the same app binary and evaluation protocol, with a separate prompt control.

The first lesson is that model capability and application performance must be measured separately. PDF support originally arrived through app changes without retraining. Auditing the successful v6 corpus later found zero dedicated PDF content targets, 162 PDF filename requests, and 108 PDF content refusals. Some successful PDF searches therefore depended on readers, routing and recovery outside the model. That history does not establish that the model independently learned reliable PDF command generation.

App improvements also helped PowerPoint. Without changing weights or prompts, they raised PPT results from 19/57 to 37/57 while preserving every non-PPT outcome. This established that reading, interpretation and ranking mattered. It also exposed a measurement trap: a successful application can conceal imperfect model instructions. Our desired direction is for the model to express intent correctly, while the application supplies general tools, execution, permissions and interface.

The focused training experiment used 17,145 legacy replay records and 222 new supervised turns from 156 requests. Each batch contained one new turn and fifteen replay turns. Across 1,332 steps, new examples received six exposures each. New targets were learned through cross entropy; replay matched a frozen v6 teacher using KL divergence multiplied by five. This controlled recipe tested repetition and preservation, rather than a wholesale restart.

Validated reference trajectories are necessary but insufficient: all 204 accepted training and validation cases executed correctly, yet unseen accuracy remained limited. Future examples need broader phrasing, composed constraints, realistic distractors and independent splits, alongside deliberate coverage of old abilities.

The loss curves improved substantially, but they measured agreement with targets and teacher probabilities. They did not directly measure whether a generated command returned the right files. A nearly correct command can still fail through one missing exclusion, an incorrect format, or an overly restrictive phrase. This gap makes executed tests essential. It does not, by itself, prove that cross entropy, KL, the coefficient five, or model capacity caused the failures.

There was measurable learning beyond the loss curves. On 48 reserved validation cases, the final checkpoint passed 22, compared with 11 for both v6 and checkpoint 666. Retention before app recovery improved from 429/472 for v6 to 447/472. Full application retention reached 467/472 against 464/472. These gains coexist with regressions elsewhere; they establish improvement on those tests without establishing a better overall model.

Our refusal hypothesis needed correction. None of checkpoint 666's 29 failed PPT searches involved an explicit refusal. At the final checkpoint, two of 26 failed PPT searches began with refusals; only one ended there. The other proceeded through recovery and still failed. Most PPT failures involved attempted searches. Preserving obsolete refusals remains a legitimate training concern, but excessive explicit refusal did not explain most observed failures.

Evaluation labels also need maintenance. One legacy test still demands refusal of searching inside PowerPoint slides, although PPTX contents are supported. Both baseline and candidate fail it, so it does not explain their difference. Its fixture contains only a canary, meaning a corrected test needs real documents and expected results. Simply changing its recorded failure into a pass would manufacture progress.

The command interface is another important factor. All ten final PPT regressions relative to v6 occurred among cases where the candidate emitted the stricter v3 plans. Those plans bypass some older corrections and preserve explicit constraints. This association identifies a useful investigation area, but cannot establish that changing the version would fix the commands. Training targets, execution semantics and evaluation expectations must agree.

Sutton's The Bitter Lesson supports investing in general learning methods instead of accumulating special rules for interpreting requests. Applying that direction means teaching complete commands and measuring their execution without semantic repair. Readers, syntax validation and sandbox enforcement remain necessary infrastructure. Removing safeguards would contribute nothing to learning.

The next research question is whether corrected joint training works better from v6 or from stock pretrained Qwen. Both should receive the same audited legacy, PPTX and new PDF data. Obsolete answers need replacement, and existing PPTX references need checking without app repair. Starting from stock Qwen retains its pretrained knowledge; it does not mean random initialization. Parameter count supplies no dependable number of remaining skills.

The completed pilot therefore leaves v6 selected. Checkpoints are saved and verified, unsafe executions remained zero, and the official holdout remains unscored. The useful outcome is a reproducible set of gains, regressions and competing explanations that can guide the next controlled experiment.