EXPERIMENTS & EVIDENCE
Notes on teaching a small, local model to find the right files. We measure progress by executing its answers and checking which tests pass.
The focused PowerPoint pilot improved reserved validation from 11/48 to 22/48, while overall development results fell from 666/703 to 654/703. What we learned about loss curves, refusal labels, replay and the work still done by the app.
Each score is executed tests passed out of tests run. All four configurations use the same frozen app binary and protocol. The prompt control runs v6 with the compact prompt used for training.
| Evaluation | Original v6 | v6 prompt control | Step 666 | Step 1,332 |
|---|---|---|---|---|
| Overall development | 666/703 | 663/703 | 647/703 | 654/703 |
| PowerPoint | 37/57 | 37/57 | 28/57 | 31/57 |
| Retention | 464/472 | 459/472 | 467/472 | 467/472 |
| Formats | 56/56 | 55/56 | 49/56 | 49/56 |
| Refusals and recovery | 60/64 | 62/64 | 59/64 | 58/64 |
| Filenames | 49/54 | 50/54 | 44/54 | 49/54 |
| All non-PPT tests | 629/646 | 626/646 | 619/646 | 623/646 |
| Reserved validation | 11/48 | 11/48 | 11/48 | 22/48 |
V6 remains selected. The candidate improved some abilities but did not meet the overall acceptance gates. The 48 reserved cases are separate from the 703 development tests; the official holdout was not scored. These are app-assisted results, and they do not establish independent model command generation.
This pilot is separate from the 312-case FindBench comparison.