Synthetic Email Gains and Executed Retrieval
Benchmark blind spots and model selection in Tofazzol
1Independent researcher · 2AI research and writing assistant
Abstract
We study the relationship between synthetic email fine-tuning and executed retrieval in Tofazzol, a 1.5-billion-parameter model used by the desktop search application Shlexball. A completed campaign added 3,000 reviewed email examples and evaluated 20 checkpoints from three training runs. Continuing from production increased accuracy on a related synthetic development set from 176/200 to 199/200, while main email accuracy remained 217/240. Equal totals concealed changes in individual outcomes: a final fresh-base checkpoint gained 19 email cases and lost 19. An earlier checkpoint corrected all 23 production sender-copying failures but introduced six topic-filter failures and substantial file-search regressions. We also examine why a historical candidate with 240/240 synthetic email accuracy was not deployed: on a separate 30-query work-mail trial, it retrieved the intended targets in 29/30 searches, compared with production’s 30/30. The benchmark lacked Sent/reply retrieval tasks, and source-data coverage constrained complete retrieval. On 100 additional generated real-cache queries, production and Guard200 passed 72/100 and 67/100 strict complete-result tests, with 91/100 reference plans establishing complete retrieval. We connect the findings to continual learning, semantic evaluation, and adaptive benchmark reuse. These observations characterize this system and its evaluation process; they do not establish a new training method or independent generalization.
Keywords: small language models; synthetic data; email retrieval; continual fine-tuning; executable evaluation; benchmark coverage.
1. Introduction
A search model can learn more examples while leaving the user’s most important failures unresolved. This becomes visible when model outputs are executed against files or mailboxes: a misspelled sender, an incorrectly bound subject, or an inaccurate size threshold can change which records are returned even when the generated plan looks reasonable.
Tofazzol translates natural-language search requests into constrained search plans. The product objective motivating this study is successful email search and retrieval, with file search as a secondary requirement. Recent training supplied more email examples, including language for retrieving replies and forwards that had already been sent. The practical question was whether the new checkpoints justified replacing production.
The completed campaign did not produce a checkpoint exceeding production’s combined search score. That result alone is insufficient for an email-first decision: one candidate did improve the email board markedly, and a historical candidate already had a perfect email score. We therefore inspect the relationship among training fit, development accuracy, individual gains and losses, real-cache retrieval, and the capabilities each evaluation actually covers.
This paper contributes a documented case study with three components: the full checkpoint trajectory of the completed campaign; an audit of specific failure patterns and missing evaluation coverage; and an account of why the highest leaderboard score did not determine deployment. A further 100-query comparison was specified in response to the small size of the original field trial. Its protocol and status are reported separately in Section 6.
2. System and measurement
The model starts from Qwen2.5-Coder-1.5B-Instruct, a member of the Qwen2.5-Coder family. [1] Tofazzol’s full fine-tuning updates 1,543,714,304 parameters. A file or email request is converted into one output line. For email, that line contains a structured mailsearch plan specifying fields such as sender, recipient, subject, topic, folder, time, ordering, and result limit. A deterministic reader executes the plan over the available Outlook cache.
Let fθ(xi) be a model’s first answer, E the fixed executor and fixture-specific scoring procedure, and yi the expected outcome. The main accuracy is:
For the work-mail comparison, an exact, complete success additionally requires the expected returned message IDs, the expected matched count, and no reported unreadable or undetermined coverage. Target retrieval is a separate, weaker measure: the intended message or target set must appear among the returned results. Finding a target does not establish that the complete answer set was found.
Token cross-entropy measures how well a model predicts reference tokens during training. It is logged as a diagnostic. Checkpoint quality in this study is assessed by executed tests, since token-level agreement does not directly determine whether a search returns the right messages. The first generated answer is scored; there is no best-of-many sampling.
3. Data, training recipes, and evaluation scope
3.1 Synthetic email additions
The campaign used a locally run teacher recorded as Gemma 4 31B to generate 3,000 email examples in ten batches of 300. The additions span 20 families, with 150 examples per family, covering sender copying, field binding, subjects, topics, calendar expressions, recipients, and retrieval of previously sent replies and forwards. Of these examples, 1,200 use Sent.
Admission included 3,000 positive execution checks, 3,000 checks against nonmatching messages, and 3,000 tokenizer-parity checks. Fixtures rejected 22,350 deliberately incorrect plan variants. Those variants are negative controls, rather than additional training rows or a count of rejected teacher outputs. The review record contains 3,157 inspected draft versions, with 72 versions flagged for repair; automated checks also caused regeneration.
The inherited corpus contained 44,071 row exposures, including 21,814 positive email exposures and 9,430 distinct normalized positive mail requests. It already included 972 Sent-folder examples. The identified language gap concerned positive retrieval of already-sent replies, forwards, or responded mail. Existing action-oriented reply/forward wording generally mapped to refusal behavior. This distinction matters: the campaign addressed a specific task interpretation gap within an existing email capability.
The initial corpus preserved the inherited rows byte-for-byte and added the 3,000 new rows, giving 47,071. The continuation and completed fresh pass also included 256 exposures of 128 previously reviewed filename examples, giving 47,327. Private mailbox contents and benchmark requests were excluded from teacher inputs and training.
3.2 Three completed runs
| Run | Initialization and recipe | Coverage and checkpoints |
|---|---|---|
| Initial fresh base | Original base model; learning rate 2 × 10−5; batch 16; microbatch 4. | Stopped at step 896 after 14,336 row exposures, including 899 new rows. Six checkpoints. This was an incomplete pass. |
| Production continuation | Production Copy400; learning rate 2 × 10−6; warmup 20; frozen-reference KL coefficient 5; replay cross-entropy coefficient 1; literal-identity weight 4. Each batch: 4 focused-file, 4 new-email, 8 replay examples. | 1,200 steps; all 3,000 additions encountered by step 750. Nine checkpoints. The reference was frozen while candidate weights were updated. |
| Complete fresh pass | Original base model; learning rate 2 × 10−5; batch 16; microbatch 4; warmup 50. | One complete pass over 47,327 rows in 2,958 steps. Five checkpoints. Intermediate scores did not truncate this pass. |
The completed fresh pass and production continuation differ in initialization, sampling, objective, and schedule. They do not isolate a causal effect of any one factor. The completed campaign also lacks a matched original-data-only control. Production itself came from successive broad, calendar/field, file, joint file/mail, and literal-copy stages. Each stage inherited the preceding stage’s weights: the released model is one final weight snapshot with a history of successive training objectives. Reusing an optimizer setting does not reproduce that sequence.
3.3 Evaluation populations
| Evaluation | Size and purpose | Interpretation limits |
|---|---|---|
| Main search board | 986 cases: 320 documents, 160 filenames, 266 media metadata, 240 email. | A repeatedly used development board. The email cases have no Sent reference plans or retrospective reply/forward/respond retrieval requests. |
| New email development | 200 synthetic cases, including 80 Sent cases. | Different identities from training, with shared generator and fixture families. |
| Training fit | 600 examples seen during candidate training. | A learning diagnostic, not a generalization test. |
| Qualification checks | 147 native calendar, 20 cache, 432 clock, 35 older private-mail cases. | The older private suite also lacks retrospective Sent/reply retrieval requests. |
| Original work-mail trial | 30 queries, with distinct target emails crosschecked against Outlook’s visible list. | Separate historical field comparison. The 20 new checkpoints were not evaluated on this suite in the completed campaign. |
| Expanded work-cache trial | 100 new generated queries, ten families, 100 distinct selected target messages. | Same frozen real cache; generated requests and cached-field expectations. Reported separately in Section 6. |
The 16 model-refusal diagnostics are excluded from the 986 search total and from selection. Application sandboxing and write-denial remain in place. The completed campaign froze its runtime, contexts, worker, inputs, and scoring rules. Some historical candidates were measured with earlier runtime/context pairings; those scores are labeled historical below.
4. Results of the completed campaign
4.1 Stronger fit and development scores did not improve the main email total
Continuation step 750 passed 599/600 training-fit examples, compared with production’s 494/600: 105 gains and no losses on that diagnostic. On the new development set, it improved from 176/200 to 199/200, an increase of 11.5 percentage points. Main email remained 217/240. The overall search total tied production at 952/986, with one fewer document pass and one additional filename pass.
The development Sent subset improved from production’s 78/80 to continuation step 600’s 80/80. That establishes performance on those synthetic cases. It does not establish that app routing, real recipient associations, or retrospective reply interpretation were repaired in deployment.
4.2 Identical totals can conceal different failures
Let G be the cases a candidate newly passes and L the cases it newly fails, relative to production. The net change is G − L, while G + L measures the number of changed pass/fail outcomes. These quantities answer different questions.
| Candidate | Both pass | Gained | Lost | Both fail | Candidate total |
|---|---|---|---|---|---|
| Continuation 750 | 215 | 2 | 2 | 21 | 217/240 |
| Fresh 2000 | 211 | 23 | 6 | 0 | 234/240 |
| Fresh 2958 | 198 | 19 | 19 | 4 | 217/240 |
The final fresh checkpoint therefore changes 15.8% of the email outcomes despite a zero net score change. A regression-sensitive deployment decision requires these paired changes. The totals alone cannot show which previously working searches would stop working.
4.3 A local email improvement accompanied substantial file regressions
Fresh step 2000 corrected all 23 production email failures, which involved altered sender spelling, and reached 234/240. It introduced six new topic-filter errors. Its file score was 645/746, compared with production’s 735/746, leaving a combined total of 879/986. The final fresh checkpoint returned to 217/240 email and finished at 912/986 overall.
An audit of the 67 media cases lost at step 2000 found that 62 used the same incorrect unitless size threshold for 2 MB requests. All 67 commands executed without harness errors. The media score declined by 66 net cases because the checkpoint also gained one media case. This concentrated error pattern is more informative than describing the result simply as a general loss of media capability.
The frozen fixtures also allowed some numerically inexact production thresholds to pass. Executing a command can establish its outcome on a fixture while leaving its broader semantics underdetermined. A test needs records near the relevant boundary to distinguish nearby thresholds. No fixture or scoring rule was changed in this campaign.
5. Why the 240/240 email candidate was not deployed
The historical leaderboard made Guard200 an attractive candidate. It had the highest combined score among the candidates discussed here and passed every case in the synthetic email board. The comparison supplied by the operator is reproduced in Table 4.
| Historical candidate | Files /746 | Email /240 | Search /986 |
|---|---|---|---|
| Guard200: best combined | 729 | 240 | 969 (98.3%) |
| Copy400: best files | 735 | 216 | 951 (96.5%) |
The later release runtime/context pairing reproduced Copy400 at 952/986, with 217/240 email and the same 735/746 file total. The weight file is the same. This one-case change is a reason to preserve runtime and context identity with every model score; it is not a new weight improvement. Guard200’s 969 is a historical figure, not a rerun under the completed campaign’s release pairing.
5.1 The deployment question concerned retrieval from a real mailbox
In a separate 30-query work-account trial, frozen first answers were executed through the sandboxed mail worker. The expected targets had been crosschecked against Outlook’s visible message list. The same model answers were replayed against the original and corrected readers. The corrected-reader results were:
| Candidate | Exact, complete /30 | Targets retrieved /30 |
|---|---|---|
| Production Copy400 | 27 | 30 |
| Guard200 | 26 | 29 |
| Derived-half blend | 25 | 29 |
| Earlier published v1 | 9 | 11 |
| Reference plans | 27 | 30 |
Guard200 lost a known real-mail target that production retrieved. Production also matched the reference plans’ 27/30 complete-result count; the remaining three cases were limited by body coverage. On the older 35-query private suite, production passed 35/35 and Guard200 34/35. These observations supported retaining the incumbent while further evidence was gathered. The six-case aggregate file advantage was secondary to the real-email retrieval evidence.
5.2 The decision was provisional and the sample was small
A one-target difference in 30 queries does not establish that production is generally better than Guard200. The queries were selected, involved one frozen environment, and did not cover Sent/reply wording or recipient constraints. Retaining a known-working incumbent was a conservative engineering decision under this evidence. It was not a statistically established ranking over real email use.
This limitation motivated the 100-query extension. Its results must be allowed to change the interpretation. A larger sample can reveal more failures, but size alone does not supply missing task coverage, independent query authorship, or missing mailbox fields.
6. Expanded comparison: 100 additional real-cache searches
The expanded protocol compares the same production Copy400 and Guard200 weights on 100 additional generated requests grounded in a frozen cache containing 2,259 messages. It selects 100 distinct target messages and 100 distinct expected result sets, excluding previously selected targets from the original 30. The original trial remains a separate historical result.
Ten families contribute ten cases each: sender name, sender address, subject phrase, sender plus subject, sender plus calendar month, subject plus calendar month, folder plus subject, Sent subject, retrospective Sent reply/forward subject, and topic mention. The folder family includes Inbox and Archive. The retrospective family includes two forward-prefixed and eight reply-prefixed Sent messages; these labels use subject prefixes and do not verify conversation relationships.
Queries and cached-field expectations are generated before either model is run. Expected result sets are calculated from the frozen metadata and text-matching contract, then crosschecked by executing reference plans through the native worker. Reference completeness and unavailable fields are recorded before inference. Each model produces one first answer under the same frozen context and runtime.
The primary measure requires both reference and candidate completeness, the exact expected ID set, and the expected matched count. Separate measures report retrieval of the selected target and agreement with the observed cached-result set. Requiring reference completeness prevents an overly narrow candidate plan from appearing to repair information absent from the cache.
Completed results. The reference plans establish complete retrieval for 91/100 cases. The remaining 9 cases have incomplete coverage under this frozen reader. All 100 have a known positive target. Strict complete-result counts use the full denominator of 100, with the shared coverage limit reported explicitly.
| Candidate | Exact, complete /100 | Selected target /100 | Exact observed set /100 |
|---|---|---|---|
| Production Copy400 | 72 | 83 | 81 |
| Guard200 | 67 | 79 | 76 |
Relative to production, Guard200 gains 0 strict complete-result cases and loses 5. For selected-target retrieval, it gains 0 and loses 4. These are paired counts on the same frozen inputs.
Production performs better on strict complete retrieval in this expansion, without a lower selected-target total. This supports retaining it within the scope of the expanded protocol. The generated and adaptively designed suite does not establish a population-wide winner.
| Family | Reference complete /10 | Production /10 | Guard200 /10 |
|---|---|---|---|
| sent retrospective | 10 | 10 | 10 |
| topic | 1 | 1 | 1 |
| sender name | 10 | 7 | 6 |
| sender address | 10 | 10 | 10 |
| subject | 10 | 10 | 10 |
| sender subject | 10 | 8 | 7 |
| sender calendar | 10 | 9 | 7 |
| subject calendar | 10 | 0 | 0 |
| folder subject | 10 | 7 | 6 |
| sent subject | 10 | 10 | 10 |
Post hoc field-binding audit. Production adds an unrequested sender filter to 10/10 subject-plus-calendar plans; Guard200 does so in 10/10. These requests use “from” before a calendar month and year. Their reference plans constrain time and subject, with no sender constraint. This audit was performed after scoring and did not change the cases or pass criteria. The shared wording makes these ten cases correlated evidence of one interpretation pattern.
Expanded protocol SHA-256: 50d6be667c3e4a2b6e3c800be44cddc48a0bf5d16938a736810aee92def2469e. Candidate answers were not observed before the protocol was frozen.
7. Concepts and related work
7.1 Synthetic data: correctness and transfer are different questions
Self-Instruct demonstrates a generate-filter-fine-tune approach to producing instruction data. [2] Our campaign likewise uses generated examples with admission checks, although a separate teacher supplies them. The distinction exposed here is between an example being executable and its training signal improving a different evaluation distribution. The checks establish specific properties of the admitted rows; they do not establish that the rows cover all operational failures.
SynAE evaluates the relationship between synthetic and real tool-agent data through several dimensions, including validity, fidelity, diversity, and downstream evaluation. [9] Its relevance is conceptual: our 200-row development set and real-cache trials measure different properties. We do not implement SynAE or claim that one synthetic score can replace its multi-axis assessment.
7.2 Continual learning: adding behavior while preserving existing behavior
Replay means revisiting earlier training examples while learning new ones. A frozen-reference penalty instead discourages changes to a previous model’s output distribution. Both are preservation mechanisms; neither guarantees unchanged executed outcomes. The continuation used replay and a KL penalty, yet exchanged two passing email cases for two previously failing cases.
Learning without Forgetting studies preservation while adding capabilities in vision, and is a useful introduction to the broader problem. [3] InsCL studies instruction-based replay and different task orders for language models. [4] These are relevant precedents, not algorithms reproduced by this campaign. The new recipe cannot be credited with a novel continual-learning mechanism on the evidence available here.
7.3 Semantic parsing: correct output depends on the meaning of a query
An email plan is a small executable interpretation of a request. A sender name copied into a topic field changes its meaning. This connects naturally to text-to-SQL evaluation. Finegan-Dollak et al. show why changing question wording or entity values can leave the underlying query pattern familiar, and why evaluation splits should address structural generalization. [5] In our setting, separate synthetic identities are useful, but shared query and fixture families still limit what a development improvement establishes.
Zhong et al. propose evaluation across distilled database test suites to better approximate semantic correctness. [6] Our media-threshold audit illustrates a related issue: two different numeric filters can return the same records on one fixture. The connection suggests adding discriminating boundary records; it does not make this paper a test-suite-accuracy implementation.
7.4 Behavioral coverage and adaptive evaluation
CheckList organizes evaluation around capabilities and test types, helping reveal behaviors hidden by a single aggregate score. [7] The missing Sent/reply family is exactly the kind of coverage gap a capability inventory should make visible. A useful inventory here separates sender copying, field binding, dates, ordering, folders, retrospective language, recipients, and source availability.
Dwork et al. formalize the problem of drawing conclusions after repeatedly reusing data to choose analyses or models. [8] Our main board was used adaptively during development. Keeping it unchanged preserves comparability, but does not restore independence. We therefore report these scores as development observations and leave reserved confirmation inputs unopened.
8. Limitations and claims supported by the evidence
Causal attribution. The completed runs change several factors at once. There is no matched original-data-only arm for them, and no matched comparison isolating the production initialization, replay, KL weight, literal-copy weight, or stage order. The observations cannot establish that adding email data caused the regressions, or that a particular preservation mechanism prevented them.
Replication and selection. Checkpoints within a run are correlated; 20 checkpoints are not 20 independent replications. There is no multi-seed estimate of variability. The repeatedly inspected boards and post hoc error audits also limit formal inference. Counts and percentage-point changes are descriptive; no claim of statistical significance is made.
Coverage and representativeness. The original 30-query trial is small and selected. The 100-query extension expands the number of targets and families, but uses generated language and one existing cache. Neither is a probability sample of email usage. In particular, a generated subject-specific retrieval request does not establish general understanding of conversational references or threads.
Shared measurement infrastructure. Execution testing depends on the reader, reference expectations, and fixtures. The original reader returned no targets for the four field-trial models; repairing folder interpretation changed the system’s observable behavior. Some media fixtures fail to distinguish inaccurate thresholds. The larger trial’s metadata export shares a decoder with the worker. These dependencies are recorded rather than treated as independent evidence.
Scope of deployment evidence. The historical work-mail trial exercised the Rust sidecar path and sandboxed worker. The expanded protocol directly invokes the frozen Python inference path and executes outputs through the sandboxed worker. It does not test current UI routing or a new app release. Results from different runtime/context pairings are kept distinct.
Reproducibility. The public artifact includes aggregate checkpoint results, figures, protocol identities, and paired transition counts. Private mail, prompts, raw predictions, corpora, and weights are withheld. The available artifact supports inspection of the arithmetic and reporting; it does not permit an outside reader to independently rerun the private-mail experiments. A publishable synthetic substitute with comparable failure modes remains future work.
9. Ongoing staged replay and the next controlled experiment
A separate experiment was authorized after the completed campaign: two arms replay the historical five-stage sequence from the original base model. One uses historical data, and the other appends the admitted 3,000 email examples. Fixed endpoints are 500 broad-training steps, 250 calendar/field steps, 600 file-refinement steps, 600 joint file/mail steps, and 400 literal-copy steps: 2,350 steps per arm. Each stage continues its own arm’s preceding weights with a fresh optimizer and the historical schedule.
This design compares added data within a staged recipe. Both arms are staged, so it does not isolate the value of staging itself. Exact sampler replay predicts exposure to 2,158 distinct additions, despite all 3,000 being present in the input corpora. Its results are outside this manuscript’s completed-training analysis.
A stronger next comparison would add a single-stage arm with controlled data exposure and optimization budget, repeat seeds, and evaluate on reserved queries with disjoint language patterns and mailbox structures. A case-based analysis should preserve both gains and losses and distinguish model errors from reader availability. Those changes would support stronger causal and generalization claims than the present case study.
10. Conclusion
The completed Tofazzol campaign learned the added training distribution well and improved a related synthetic development set, while leaving the main email total unchanged at its best combined checkpoint. Paired analysis exposed changes hidden by those totals; a different checkpoint improved sender copying while introducing topic and file failures. Historical deployment evidence further showed why a perfect synthetic email score did not settle a real-mail retrieval decision.
The practical lesson is to preserve the distinction among training fit, development performance, executed behavior, coverage, and operational retrieval. The broader 100-query comparison extends the evidence under an explicit protocol. The central research contribution at this stage is a transparent empirical account of those distinctions and their consequences for choosing a compact search model.
Appendix A. All completed checkpoints
These tables are generated from saved summary files and checked against per-case pass counts. Only the three completed new-email runs are included. “Initial” is the interrupted fresh run; “Continued” starts from production; “Full pass” completes one fresh-base pass. Production is the same-run baseline.
| Run | Step | Docs /320 | Names /160 | Media /266 | Email /240 | Total /986 |
|---|---|---|---|---|---|---|
| Production | — | 318 | 155 | 262 | 217 | 952 |
| Initial | 250 | 271 | 152 | 257 | 213 | 893 |
| Initial | 400 | 206 | 66 | 245 | 231 | 748 |
| Initial | 500 | 227 | 149 | 214 | 230 | 820 |
| Initial | 600 | 294 | 111 | 202 | 223 | 830 |
| Initial | 750 | 270 | 150 | 247 | 196 | 863 |
| Initial | 896 | 297 | 150 | 199 | 219 | 865 |
| Continued | 100 | 317 | 156 | 261 | 217 | 951 |
| Continued | 200 | 317 | 155 | 262 | 216 | 950 |
| Continued | 300 | 318 | 154 | 261 | 215 | 948 |
| Continued | 400 | 317 | 154 | 262 | 217 | 950 |
| Continued | 600 | 317 | 156 | 262 | 215 | 950 |
| Continued | 750 | 317 | 156 | 262 | 217 | 952 |
| Continued | 900 | 315 | 156 | 262 | 217 | 950 |
| Continued | 1000 | 315 | 156 | 262 | 217 | 950 |
| Continued | 1200 | 316 | 156 | 262 | 217 | 951 |
| Full pass | 1000 | 301 | 149 | 252 | 214 | 916 |
| Full pass | 1500 | 289 | 154 | 257 | 216 | 916 |
| Full pass | 2000 | 296 | 153 | 196 | 234 | 879 |
| Full pass | 2500 | 302 | 153 | 234 | 224 | 913 |
| Full pass | 2958 | 299 | 153 | 243 | 217 | 912 |
| Run | Step | Calendar /147 | Cache /20 | Clock /432 | Private /35 | New dev /200 |
|---|---|---|---|---|---|---|
| Production | — | 147 | 20 | 432 | 35 | 176 |
| Initial | 250 | 131 | 15 | 352 | 34 | 183 |
| Initial | 400 | 144 | 19 | 416 | 35 | 164 |
| Initial | 500 | 144 | 17 | 416 | 34 | 180 |
| Initial | 600 | 145 | 19 | 432 | 35 | 174 |
| Initial | 750 | 135 | 20 | 304 | 35 | 174 |
| Initial | 896 | 145 | 20 | 432 | 35 | 166 |
| Continued | 100 | 147 | 19 | 432 | 35 | 194 |
| Continued | 200 | 147 | 20 | 432 | 35 | 198 |
| Continued | 300 | 147 | 20 | 432 | 35 | 198 |
| Continued | 400 | 147 | 20 | 432 | 35 | 197 |
| Continued | 600 | 147 | 20 | 432 | 35 | 199 |
| Continued | 750 | 147 | 20 | 432 | 35 | 199 |
| Continued | 900 | 147 | 20 | 432 | 35 | 199 |
| Continued | 1000 | 147 | 20 | 432 | 35 | 199 |
| Continued | 1200 | 147 | 20 | 432 | 35 | 199 |
| Full pass | 1000 | 135 | 19 | 432 | 35 | 193 |
| Full pass | 1500 | 135 | 16 | 336 | 35 | 181 |
| Full pass | 2000 | 147 | 20 | 432 | 35 | 184 |
| Full pass | 2500 | 147 | 20 | 432 | 35 | 189 |
| Full pass | 2958 | 147 | 20 | 432 | 35 | 190 |
Appendix B. Concepts in plain language
| Concept | Meaning in this study |
|---|---|
| Checkpoint | A saved weight snapshot at a particular training step. Later snapshots can perform worse on a chosen task. |
| Full fine-tuning | Updating the model’s parameters, rather than only a small adapter. The continuation’s frozen reference is a separate comparison model. |
| Cross-entropy | A penalty for predicting reference tokens poorly. It is useful during optimization but does not directly count successful searches. |
| Replay | Mixing earlier examples into later training to retain earlier behavior. |
| KL regularization | A penalty for changing an output distribution relative to a reference. Its coefficient controls its influence in the training objective. |
| Distribution shift | A change in the kinds of requests, entities, data structures, or available fields encountered at evaluation or deployment. |
| Semantic parsing | Turning a language request into an executable representation of its meaning, such as an email filter or SQL query. |
| Paired comparison | Checking which exact cases each model gains and loses on the same inputs. |
| Coverage ceiling | A measured limit on complete retrieval imposed by available source fields under the reference plans, rather than a proven limit on model learning. |
| Adaptive development | Using earlier evaluation results to guide later choices. Repeatedly consulted test cases become part of the development process. |
Appendix C. Evidence identities and data availability
All reported training checkpoints were recovered privately and associated with completion and inference identities. Each new full-model weight file has 3,087,467,144 bytes. Hashes below identify the files and evaluation protocols used for the completed campaign; a hash records identity, rather than proving a result correct.
| Artifact | SHA-256 |
|---|---|
| Production Copy400 weights | f69c89242cdf8de3662d89d09ed0d61441cfa908afb423886fd4574f550cf8d4 |
| Continuation 750 weights | e0a73bb829ae7b3c26cf830ac4fc37f0cbead8080bda263e21c459716db97955 |
| Fresh 2958 weights | 27d8f0cc28ed2bebac9ac93f66121eb40e91138a306586322fcfbc148c853452 |
| Main evaluation protocol | 62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2 |
| Qualification protocol | 869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc |
The companion aggregate results file contains only allowlisted counts, recipes, model identities, and source-artifact hashes. Exportable figures are provided as Figure 1 PDF, Figure 2 PDF, and Figure 3 PDF. All three figures are embedded in this HTML, so the manuscript itself can be read offline.
Private data remain private. This paper contains no training rows, mailbox messages, real query text, raw model predictions, per-case identifiers, or weight tensors. No private data are linked for download. The expanded comparison uses real cached mail locally; only its aggregate results can appear here.
Inspect the embedded aggregate data
{
"admission": {
"by_family": {
"calendar_composition": 150,
"calendar_person_contrast": 150,
"field_subject_topic": 150,
"identity_in_subject": 150,
"inbox_reply_subject": 150,
"retrieval_action_contrast": 150,
"sender_address": 150,
"sender_calendar": 150,
"sender_either": 150,
"sender_name": 150,
"sender_order": 150,
"sender_subject": 150,
"sender_topic": 150,
"sent_calendar_subject": 150,
"sent_forward_subject": 150,
"sent_recipient": 150,
"sent_reply_subject": 150,
"sent_reply_topic": 150,
"sent_topic_organisation": 150,
"sent_what_did": 150
},
"empty_checks": 3000,
"manual_rejected_versions": 72,
"manual_versions_read": 3157,
"new_rows": 3000,
"positive_checks": 3000,
"tokenizer_parity_rows": 3000,
"wrong_plans_rejected": 22350
},
"date": "2026-11-08",
"expanded_real_cache": {
"independent_confirmation": false,
"limitations": [
"One existing mailbox snapshot; not a random sample of user traffic.",
"Queries are generated and structured; prior findings informed the chosen families.",
"Cached body text is partial; unavailable full text is reported separately.",
"Sent reply/forward labels use subject prefixes, not verified thread relationships.",
"Recipients, attachments, app routing and live Outlook UI are not evaluated.",
"The native reader and its metadata export share a decoding implementation."
],
"models": [
{
"actual_rust_ipc": false,
"actual_sandboxed_worker": true,
"app_routing_tested": false,
"by_family": {
"folder_subject": {
"observed_set_exact": 7,
"passed": 7,
"reference_complete": 10,
"target_retrieved": 7,
"total": 10
},
"sender_address": {
"observed_set_exact": 10,
"passed": 10,
"reference_complete": 10,
"target_retrieved": 10,
"total": 10
},
"sender_calendar": {
"observed_set_exact": 9,
"passed": 9,
"reference_complete": 10,
"target_retrieved": 10,
"total": 10
},
"sender_name": {
"observed_set_exact": 7,
"passed": 7,
"reference_complete": 10,
"target_retrieved": 7,
"total": 10
},
"sender_subject": {
"observed_set_exact": 8,
"passed": 8,
"reference_complete": 10,
"target_retrieved": 9,
"total": 10
},
"sent_retrospective": {
"observed_set_exact": 10,
"passed": 10,
"reference_complete": 10,
"target_retrieved": 10,
"total": 10
},
"sent_subject": {
"observed_set_exact": 10,
"passed": 10,
"reference_complete": 10,
"target_retrieved": 10,
"total": 10
},
"subject": {
"observed_set_exact": 10,
"passed": 10,
"reference_complete": 10,
"target_retrieved": 10,
"total": 10
},
"subject_calendar": {
"observed_set_exact": 0,
"passed": 0,
"reference_complete": 10,
"target_retrieved": 0,
"total": 10
},
"topic": {
"observed_set_exact": 10,
"passed": 1,
"reference_complete": 1,
"target_retrieved": 10,
"total": 10
}
},
"generated_queries_on_real_cache": true,
"independent_confirmation": false,
"model": "production-copy400",
"observed_set_exact": 81,
"passed": 72,
"protocol_sha256": "50d6be667c3e4a2b6e3c800be44cddc48a0bf5d16938a736810aee92def2469e",
"reference_complete": 91,
"target_retrieved": 83,
"total": 100,
"weights_sha256": "f69c89242cdf8de3662d89d09ed0d61441cfa908afb423886fd4574f550cf8d4"
},
{
"actual_rust_ipc": false,
"actual_sandboxed_worker": true,
"app_routing_tested": false,
"by_family": {
"folder_subject": {
"observed_set_exact": 6,
"passed": 6,
"reference_complete": 10,
"target_retrieved": 6,
"total": 10
},
"sender_address": {
"observed_set_exact": 10,
"passed": 10,
"reference_complete": 10,
"target_retrieved": 10,
"total": 10
},
"sender_calendar": {
"observed_set_exact": 7,
"passed": 7,
"reference_complete": 10,
"target_retrieved": 8,
"total": 10
},
"sender_name": {
"observed_set_exact": 6,
"passed": 6,
"reference_complete": 10,
"target_retrieved": 6,
"total": 10
},
"sender_subject": {
"observed_set_exact": 7,
"passed": 7,
"reference_complete": 10,
"target_retrieved": 9,
"total": 10
},
"sent_retrospective": {
"observed_set_exact": 10,
"passed": 10,
"reference_complete": 10,
"target_retrieved": 10,
"total": 10
},
"sent_subject": {
"observed_set_exact": 10,
"passed": 10,
"reference_complete": 10,
"target_retrieved": 10,
"total": 10
},
"subject": {
"observed_set_exact": 10,
"passed": 10,
"reference_complete": 10,
"target_retrieved": 10,
"total": 10
},
"subject_calendar": {
"observed_set_exact": 0,
"passed": 0,
"reference_complete": 10,
"target_retrieved": 0,
"total": 10
},
"topic": {
"observed_set_exact": 10,
"passed": 1,
"reference_complete": 1,
"target_retrieved": 10,
"total": 10
}
},
"generated_queries_on_real_cache": true,
"independent_confirmation": false,
"model": "guard200",
"observed_set_exact": 76,
"passed": 67,
"protocol_sha256": "50d6be667c3e4a2b6e3c800be44cddc48a0bf5d16938a736810aee92def2469e",
"reference_complete": 91,
"target_retrieved": 79,
"total": 100,
"weights_sha256": "d6afb87423e0910df586e3e66c35dff95afabebd8d468fda1deacd6bc7bafce8"
}
],
"original_30_kept_separate": true,
"paired": {
"observed_set_exact": {
"both_fail": 19,
"both_pass": 76,
"guard_gains": 0,
"guard_losses": 5
},
"passed": {
"both_fail": 28,
"both_pass": 67,
"guard_gains": 0,
"guard_losses": 5
},
"target_retrieved": {
"both_fail": 17,
"both_pass": 79,
"guard_gains": 0,
"guard_losses": 4
}
},
"post_hoc_audit": {
"guard200": {
"subject_calendar_cases": 10,
"unrequested_sender_filter": 10
},
"production-copy400": {
"subject_calendar_cases": 10,
"unrequested_sender_filter": 10
}
},
"protocol_sha256": "50d6be667c3e4a2b6e3c800be44cddc48a0bf5d16938a736810aee92def2469e",
"reference_complete": 91,
"total": 100,
"unique_target_messages": 100
},
"field_suite": {
"published-v1": {
"app_sha256": "eb853a8c53a8b5bf706135cce53b28775b68a1b53ae20af0fd5960675d5ac95d",
"passed": 9,
"targets_retrieved": 11,
"total": 30
},
"v2-app-validated": {
"app_sha256": "eb853a8c53a8b5bf706135cce53b28775b68a1b53ae20af0fd5960675d5ac95d",
"passed": 25,
"targets_retrieved": 29,
"total": 30
},
"v2-best-combined": {
"app_sha256": "eb853a8c53a8b5bf706135cce53b28775b68a1b53ae20af0fd5960675d5ac95d",
"passed": 26,
"targets_retrieved": 29,
"total": 30
},
"v2-best-files": {
"app_sha256": "eb853a8c53a8b5bf706135cce53b28775b68a1b53ae20af0fd5960675d5ac95d",
"passed": 27,
"targets_retrieved": 30,
"total": 30
}
},
"independent_confirmation": false,
"media_audit": {
"candidate_repeats_unitless_size_20960": 62,
"media_lost": 67
},
"recipes": {
"continued": {
"batch": 16,
"epochs": 1,
"lr": 2e-06,
"max_steps": 1200,
"microbatch": 4,
"seed": 20261105,
"warmup": 20
},
"fresh": {
"batch": 16,
"epochs": 1,
"lr": 2e-05,
"max_steps": 2958,
"microbatch": 4,
"seed": 20261105,
"warmup": 50
}
},
"rows": [
{
"arm": "Production",
"file_passed": 735,
"label": "baseline",
"order": 0,
"paired_email": {
"both_fail": 23,
"both_pass": 217,
"gained": 0,
"lost": 0
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
318,
320
],
"filenames": [
155,
160
],
"media": [
262,
266
],
"new_mail_development": [
176,
200
],
"outlook": [
217,
240
]
},
"search_passed": 952,
"step": null,
"weights_sha256": "f69c89242cdf8de3662d89d09ed0d61441cfa908afb423886fd4574f550cf8d4"
},
{
"arm": "Initial",
"file_passed": 680,
"label": "tofazzol-v1-gemma-email-3000-fresh-20261108-checkpoint-00250",
"order": 1,
"paired_email": {
"both_fail": 9,
"both_pass": 199,
"gained": 14,
"lost": 18
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
15,
20
],
"calendar": [
131,
147
],
"clock": [
352,
432
],
"private": [
34,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
271,
320
],
"filenames": [
152,
160
],
"media": [
257,
266
],
"new_mail_development": [
183,
200
],
"outlook": [
213,
240
]
},
"search_passed": 893,
"step": 250,
"weights_sha256": "01284ee6233a13d57e2f02bd519012ee13825eee001928dd27ca26aab8b13540"
},
{
"arm": "Initial",
"file_passed": 517,
"label": "tofazzol-v1-gemma-email-3000-fresh-20261108-checkpoint-00400",
"order": 1,
"paired_email": {
"both_fail": 6,
"both_pass": 214,
"gained": 17,
"lost": 3
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
19,
20
],
"calendar": [
144,
147
],
"clock": [
416,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
206,
320
],
"filenames": [
66,
160
],
"media": [
245,
266
],
"new_mail_development": [
164,
200
],
"outlook": [
231,
240
]
},
"search_passed": 748,
"step": 400,
"weights_sha256": "143d71db3e28b45874354a2ca428d09497ef2d458b2fe954bac05434ded9b065"
},
{
"arm": "Initial",
"file_passed": 590,
"label": "tofazzol-v1-gemma-email-3000-fresh-20261108-checkpoint-00500",
"order": 1,
"paired_email": {
"both_fail": 6,
"both_pass": 213,
"gained": 17,
"lost": 4
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
17,
20
],
"calendar": [
144,
147
],
"clock": [
416,
432
],
"private": [
34,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
227,
320
],
"filenames": [
149,
160
],
"media": [
214,
266
],
"new_mail_development": [
180,
200
],
"outlook": [
230,
240
]
},
"search_passed": 820,
"step": 500,
"weights_sha256": "7cd1241d3ee928468db2c18ed7c85db751998316c72fc2d77161ee09f3d4a49a"
},
{
"arm": "Initial",
"file_passed": 607,
"label": "tofazzol-v1-gemma-email-3000-fresh-20261108-checkpoint-00600",
"order": 1,
"paired_email": {
"both_fail": 4,
"both_pass": 204,
"gained": 19,
"lost": 13
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
19,
20
],
"calendar": [
145,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
294,
320
],
"filenames": [
111,
160
],
"media": [
202,
266
],
"new_mail_development": [
174,
200
],
"outlook": [
223,
240
]
},
"search_passed": 830,
"step": 600,
"weights_sha256": "7f5b5150484228575f7dcd7435c671ffd075f76043af52bd6c4def29aa21665a"
},
{
"arm": "Initial",
"file_passed": 667,
"label": "tofazzol-v1-gemma-email-3000-fresh-20261108-checkpoint-00750",
"order": 1,
"paired_email": {
"both_fail": 11,
"both_pass": 184,
"gained": 12,
"lost": 33
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
135,
147
],
"clock": [
304,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
270,
320
],
"filenames": [
150,
160
],
"media": [
247,
266
],
"new_mail_development": [
174,
200
],
"outlook": [
196,
240
]
},
"search_passed": 863,
"step": 750,
"weights_sha256": "3192f237fd5ce62e4ab0ed26858794428487ca0ae152cb78a1d460377bddbba2"
},
{
"arm": "Initial",
"file_passed": 646,
"label": "tofazzol-v1-gemma-email-3000-fresh-20261108-checkpoint-00896",
"order": 1,
"paired_email": {
"both_fail": 3,
"both_pass": 199,
"gained": 20,
"lost": 18
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
145,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
297,
320
],
"filenames": [
150,
160
],
"media": [
199,
266
],
"new_mail_development": [
166,
200
],
"outlook": [
219,
240
]
},
"search_passed": 865,
"step": 896,
"weights_sha256": "8b7290b70c1cbabfc30b4155601a4d0d618e3adb0e68a5d727264c150e8735ee"
},
{
"arm": "Continued",
"file_passed": 734,
"label": "tofazzol-v1-gemma-email-3000-filename-focus-release-20261108-checkpoint-00100",
"order": 2,
"paired_email": {
"both_fail": 21,
"both_pass": 215,
"gained": 2,
"lost": 2
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
19,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
317,
320
],
"filenames": [
156,
160
],
"media": [
261,
266
],
"new_mail_development": [
194,
200
],
"outlook": [
217,
240
]
},
"search_passed": 951,
"step": 100,
"weights_sha256": "8fc7814a51b48db257b41fb9f06843d71ccef9986df0b1f4a9054486025d512b"
},
{
"arm": "Continued",
"file_passed": 734,
"label": "tofazzol-v1-gemma-email-3000-filename-focus-release-20261108-checkpoint-00200",
"order": 2,
"paired_email": {
"both_fail": 20,
"both_pass": 213,
"gained": 3,
"lost": 4
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
317,
320
],
"filenames": [
155,
160
],
"media": [
262,
266
],
"new_mail_development": [
198,
200
],
"outlook": [
216,
240
]
},
"search_passed": 950,
"step": 200,
"weights_sha256": "2a89107e0eee005f21dc8eac211c939c0bbc1ecf19416d2ea57970d9feffae9c"
},
{
"arm": "Continued",
"file_passed": 733,
"label": "tofazzol-v1-gemma-email-3000-filename-focus-release-20261108-checkpoint-00300",
"order": 2,
"paired_email": {
"both_fail": 21,
"both_pass": 213,
"gained": 2,
"lost": 4
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
318,
320
],
"filenames": [
154,
160
],
"media": [
261,
266
],
"new_mail_development": [
198,
200
],
"outlook": [
215,
240
]
},
"search_passed": 948,
"step": 300,
"weights_sha256": "12cd53f8c63f55283061c5e96d6902c9f53c66ecba5572a2c1beea5989626ce0"
},
{
"arm": "Continued",
"file_passed": 733,
"label": "tofazzol-v1-gemma-email-3000-filename-focus-release-20261108-checkpoint-00400",
"order": 2,
"paired_email": {
"both_fail": 21,
"both_pass": 215,
"gained": 2,
"lost": 2
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
317,
320
],
"filenames": [
154,
160
],
"media": [
262,
266
],
"new_mail_development": [
197,
200
],
"outlook": [
217,
240
]
},
"search_passed": 950,
"step": 400,
"weights_sha256": "6b7243c6298b17cd06472cf0ba6b9150418cf998b3f202a973a16441f4f8b39c"
},
{
"arm": "Continued",
"file_passed": 735,
"label": "tofazzol-v1-gemma-email-3000-filename-focus-release-20261108-checkpoint-00600",
"order": 2,
"paired_email": {
"both_fail": 21,
"both_pass": 213,
"gained": 2,
"lost": 4
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
317,
320
],
"filenames": [
156,
160
],
"media": [
262,
266
],
"new_mail_development": [
199,
200
],
"outlook": [
215,
240
]
},
"search_passed": 950,
"step": 600,
"weights_sha256": "a212bc25f3ff984ee19f9f40691179c8fd94be1256f757ef56a7c118b41e30c6"
},
{
"arm": "Continued",
"file_passed": 735,
"label": "tofazzol-v1-gemma-email-3000-filename-focus-release-20261108-checkpoint-00750",
"order": 2,
"paired_email": {
"both_fail": 21,
"both_pass": 215,
"gained": 2,
"lost": 2
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
317,
320
],
"filenames": [
156,
160
],
"media": [
262,
266
],
"new_mail_development": [
199,
200
],
"outlook": [
217,
240
]
},
"search_passed": 952,
"step": 750,
"weights_sha256": "e0a73bb829ae7b3c26cf830ac4fc37f0cbead8080bda263e21c459716db97955"
},
{
"arm": "Continued",
"file_passed": 733,
"label": "tofazzol-v1-gemma-email-3000-filename-focus-release-20261108-checkpoint-00900",
"order": 2,
"paired_email": {
"both_fail": 21,
"both_pass": 215,
"gained": 2,
"lost": 2
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
315,
320
],
"filenames": [
156,
160
],
"media": [
262,
266
],
"new_mail_development": [
199,
200
],
"outlook": [
217,
240
]
},
"search_passed": 950,
"step": 900,
"weights_sha256": "8fd8969e0d3e64cc36f85227b90581103369508bb9da28e50f743c2becf1086c"
},
{
"arm": "Continued",
"file_passed": 733,
"label": "tofazzol-v1-gemma-email-3000-filename-focus-release-20261108-checkpoint-01000",
"order": 2,
"paired_email": {
"both_fail": 21,
"both_pass": 215,
"gained": 2,
"lost": 2
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
315,
320
],
"filenames": [
156,
160
],
"media": [
262,
266
],
"new_mail_development": [
199,
200
],
"outlook": [
217,
240
]
},
"search_passed": 950,
"step": 1000,
"weights_sha256": "b035f3a3cb5afed90aae2bf4f5ba0ac5c4caef703a71260fe78b5289f15c3a3a"
},
{
"arm": "Continued",
"file_passed": 734,
"label": "tofazzol-v1-gemma-email-3000-filename-focus-release-20261108-checkpoint-01200",
"order": 2,
"paired_email": {
"both_fail": 21,
"both_pass": 215,
"gained": 2,
"lost": 2
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
316,
320
],
"filenames": [
156,
160
],
"media": [
262,
266
],
"new_mail_development": [
199,
200
],
"outlook": [
217,
240
]
},
"search_passed": 951,
"step": 1200,
"weights_sha256": "bee831abf54ae6dcaffdf8257e95c8592e7f59548c16aa90f8ccdea72149f5a2"
},
{
"arm": "Full pass",
"file_passed": 702,
"label": "tofazzol-v1-gemma-email-3000-full-pass-20261108-checkpoint-01000",
"order": 3,
"paired_email": {
"both_fail": 5,
"both_pass": 196,
"gained": 18,
"lost": 21
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
19,
20
],
"calendar": [
135,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
301,
320
],
"filenames": [
149,
160
],
"media": [
252,
266
],
"new_mail_development": [
193,
200
],
"outlook": [
214,
240
]
},
"search_passed": 916,
"step": 1000,
"weights_sha256": "3f473ccf791480d4bb55c620c06c8cced4577b5bf0d7f83933797fedb22f8d30"
},
{
"arm": "Full pass",
"file_passed": 700,
"label": "tofazzol-v1-gemma-email-3000-full-pass-20261108-checkpoint-01500",
"order": 3,
"paired_email": {
"both_fail": 5,
"both_pass": 198,
"gained": 18,
"lost": 19
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
16,
20
],
"calendar": [
135,
147
],
"clock": [
336,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
289,
320
],
"filenames": [
154,
160
],
"media": [
257,
266
],
"new_mail_development": [
181,
200
],
"outlook": [
216,
240
]
},
"search_passed": 916,
"step": 1500,
"weights_sha256": "af3e55aa0081ecf366c50b5722b5a17d0bc9516fd3b6bcfe05e794b4f867c6d6"
},
{
"arm": "Full pass",
"file_passed": 645,
"label": "tofazzol-v1-gemma-email-3000-full-pass-20261108-checkpoint-02000",
"order": 3,
"paired_email": {
"both_fail": 0,
"both_pass": 211,
"gained": 23,
"lost": 6
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
296,
320
],
"filenames": [
153,
160
],
"media": [
196,
266
],
"new_mail_development": [
184,
200
],
"outlook": [
234,
240
]
},
"search_passed": 879,
"step": 2000,
"weights_sha256": "30feec3007770a0873fae1390b9d594a77197d82198b1951b6ad5ce86ce078d1"
},
{
"arm": "Full pass",
"file_passed": 689,
"label": "tofazzol-v1-gemma-email-3000-full-pass-20261108-checkpoint-02500",
"order": 3,
"paired_email": {
"both_fail": 3,
"both_pass": 204,
"gained": 20,
"lost": 13
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
302,
320
],
"filenames": [
153,
160
],
"media": [
234,
266
],
"new_mail_development": [
189,
200
],
"outlook": [
224,
240
]
},
"search_passed": 913,
"step": 2500,
"weights_sha256": "248908d9c533f28d3e6b900eb7c69c58008fa2c0bb54780228e4ffde6f1e2866"
},
{
"arm": "Full pass",
"file_passed": 695,
"label": "tofazzol-v1-gemma-email-3000-full-pass-20261108-checkpoint-02958",
"order": 3,
"paired_email": {
"both_fail": 4,
"both_pass": 198,
"gained": 19,
"lost": 19
},
"protocol_sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2",
"qualification": {
"cache": [
20,
20
],
"calendar": [
147,
147
],
"clock": [
432,
432
],
"private": [
35,
35
]
},
"qualification_protocol_sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc",
"scores": {
"documents": [
299,
320
],
"filenames": [
153,
160
],
"media": [
243,
266
],
"new_mail_development": [
190,
200
],
"outlook": [
217,
240
]
},
"search_passed": 912,
"step": 2958,
"weights_sha256": "27d8f0cc28ed2bebac9ac93f66121eb40e91138a306586322fcfbc148c853452"
}
],
"schema": "tofazzol-email-empirical-paper-v1",
"scope": "Three completed new-email training runs; ongoing staged replay excluded.",
"sent_development": {
"cases": 80,
"models": {
"baseline": {
"passed": 78,
"total": 80
},
"tofazzol-v1-gemma-email-3000-filename-focus-release-20261108-checkpoint-00600": {
"passed": 80,
"total": 80
}
}
},
"source_artifact_hashes": {
"evidence-001": {
"artifact": "admission-final-summary.json",
"sha256": "4311ea93405bc15f1f83667886fca0e56c1e6e9eb72af8bc1c9ce3c55fdc9468"
},
"evidence-002": {
"artifact": "comparison.json",
"sha256": "c35907ae6be463b7197db6dc393267397d31680044eeeac79039f4c6bb25e330"
},
"evidence-003": {
"artifact": "results.json",
"sha256": "afed61305c1b7106a76e0b8e3b46e76653d62645117485b47a870ac09a02f727"
},
"evidence-004": {
"artifact": "summary.json",
"sha256": "c86c194aab0c622dfdadf4954622416c5afc74ceaf4b72d86046f45a02c7c94e"
},
"evidence-005": {
"artifact": "protocol.json",
"sha256": "62ae49b0fe89dd75b0e803863d4f51152284b7e021a79f52fa427819e55de7f2"
},
"evidence-006": {
"artifact": "results.json",
"sha256": "ec5029ec7928015de16982405e632883d85c4db7f8b7c1b2bf7de655423ef34b"
},
"evidence-007": {
"artifact": "summary.json",
"sha256": "18c40d17525eb244ddf539cac32b4b4653c899389a4bd43237b8da1bdff772af"
},
"evidence-008": {
"artifact": "results.json",
"sha256": "664a057468add96c03421c1bf303b3503369c5a54e54f937ff715de817b2ee11"
},
"evidence-009": {
"artifact": "summary.json",
"sha256": "4db50ab8e6dc795d6d13eb03d9535e04bf4409354b65cc8b233213801f56d6be"
},
"evidence-010": {
"artifact": "results.json",
"sha256": "95e5f7b857c2147edb4faaf839e99477bf91213ceb50774008e083786be0d00f"
},
"evidence-011": {
"artifact": "summary.json",
"sha256": "e18c4b5635467d71a03919e9823720d5afacd2acdabe9c3ec1740c219de8ed83"
},
"evidence-012": {
"artifact": "results.json",
"sha256": "149ce534eab299533d51206403a7d01e9a417242abcacfd0fdb174d0f275c3d5"
},
"evidence-013": {
"artifact": "summary.json",
"sha256": "79968509bbcb73629324ba0743e0544cfe98fdff31492a8ff8067e4cd3a34cc8"
},
"evidence-014": {
"artifact": "results.json",
"sha256": "3e0c684c5b261f2636eed56faf0732f075db2f0386d62e933961e7c097c144e9"
},
"evidence-015": {
"artifact": "summary.json",
"sha256": "9ab48f48f5fb3542b6c15fadb1f2cda32e07292417e98786bccea75b76f7b71e"
},
"evidence-016": {
"artifact": "results.json",
"sha256": "21cf339e12c82be21c99c30ecf872746301038105dc43c0bd376b607cc64a2da"
},
"evidence-017": {
"artifact": "summary.json",
"sha256": "e8e981f1993ba1e621c25aa1f79d9e3eca4041453ffc59be879756dd221d4d88"
},
"evidence-018": {
"artifact": "results.json",
"sha256": "e0c1b06c9cbaca90fa6d58809219010e1fb3ac640e3f968f87b7d4c8f16d4ad9"
},
"evidence-019": {
"artifact": "summary.json",
"sha256": "ce28fd9db9def3b1b2952fa5526964104e73a4677d4ba4bf46bfb9090c8cac4e"
},
"evidence-020": {
"artifact": "results.json",
"sha256": "54a602020500164574ca0f6404b77a8555881280f7eb173ac149d3694e081097"
},
"evidence-021": {
"artifact": "summary.json",
"sha256": "3b99f80796bc206cc7b6b377749aa25ff8195016a58ef4461a3274553f40b7e9"
},
"evidence-022": {
"artifact": "results.json",
"sha256": "c41493b3fe8efc78542e0ec24e0c55abcfebe1a2b265d7db9c0c693b7ea498fa"
},
"evidence-023": {
"artifact": "summary.json",
"sha256": "2d135bdd498a10963531f0efd6e01d69c35f891fb8787a916cfb9db855d0cf8d"
},
"evidence-024": {
"artifact": "results.json",
"sha256": "7999c70da9baee6a9e787645b6078f76fc00c971c11b5840b44366ddb4cae27c"
},
"evidence-025": {
"artifact": "summary.json",
"sha256": "42cf9cb4041cb77253ec252b878e259c53635384b633f5296a2ae4e71b8d9f27"
},
"evidence-026": {
"artifact": "results.json",
"sha256": "73d43084b968db51c0310ffdc18857f51bebe6bd72f5a8eff23f75289d809ccc"
},
"evidence-027": {
"artifact": "summary.json",
"sha256": "c924ad1127aeb20699c7dce4a482a2cd09c5b706289f7f4bc5c6b46ea9c33bd6"
},
"evidence-028": {
"artifact": "results.json",
"sha256": "00dc06f567b61b9445a2d72d7d72e923424aace5ad23bf8ea7ff3a2d73b769a1"
},
"evidence-029": {
"artifact": "summary.json",
"sha256": "c2202ddad965848e1edef0de6d98435245c2ed4880a706f26d3ef6f8025a2bc6"
},
"evidence-030": {
"artifact": "results.json",
"sha256": "3000a4542bd1ef219bd59204c83bafb06de415810de856f0f3eaa6b722ab77d2"
},
"evidence-031": {
"artifact": "summary.json",
"sha256": "49ca4a3e21f8b56d00e68a2f053248017808e7384c269dad8a76149e04a19044"
},
"evidence-032": {
"artifact": "results.json",
"sha256": "0e827bd924e7e0104e6cc7b049184ee144e4fcda7159ccda0c431c8b30d0bbab"
},
"evidence-033": {
"artifact": "summary.json",
"sha256": "e0716977d4c1e677ad39b464490dae327a0a545a88655f494aef0ab49a81e2d1"
},
"evidence-034": {
"artifact": "results.json",
"sha256": "0daf39e3df489216b5a6dac4618d1b71c29a7c123d198203702c2d3248fa1858"
},
"evidence-035": {
"artifact": "summary.json",
"sha256": "e62ba2bf1c038de8042430c80aca8a685966711c3bec8f22a3f1d3e1a08b3cf6"
},
"evidence-036": {
"artifact": "results.json",
"sha256": "5ce66542de24985d886e77e98b49e5ac635fa54be3fe7ac133fb0859de080542"
},
"evidence-037": {
"artifact": "summary.json",
"sha256": "54a8270e1f2f39b2bb9b346018d2f5f70527f41020a11db46076852ee9c2c298"
},
"evidence-038": {
"artifact": "results.json",
"sha256": "258db8938c80c7da64167f43a9affab136a68ee4ca28ce8aa3b2b5482b688f2d"
},
"evidence-039": {
"artifact": "summary.json",
"sha256": "aae5fe5108699dce1535183cf4bd1373c2e6c39a6c6ae3aa1221d09c6c9297af"
},
"evidence-040": {
"artifact": "results.json",
"sha256": "22f651f3840ab65cc9cb4fed09def9f2a503b0f3b0c0593c0e5336ab6f53285b"
},
"evidence-041": {
"artifact": "summary.json",
"sha256": "f2834283a9118dfc784c78db2eecdce1e5b8568da0bb410be416b593de66b321"
},
"evidence-042": {
"artifact": "results.json",
"sha256": "7cdf16b3f092c3d404419543dcaae1e812503d427ddb3ba6f208622b984c1840"
},
"evidence-043": {
"artifact": "summary.json",
"sha256": "e3a4e281e0724355088190c04ed954204012ce8917477bc242bb9e9e3b44e304"
},
"evidence-044": {
"artifact": "results.json",
"sha256": "a87546f3324e8aacef62f108d9f037d65860e8a626babe4abed251aefe5af364"
},
"evidence-045": {
"artifact": "summary.json",
"sha256": "630145a048b7ca2da8b2be478b563692020437a5866ba1e83d8b115e53627c68"
},
"evidence-046": {
"artifact": "full-pass-2000-media-error-audit.json",
"sha256": "cfdefc1ae1c84d95bc01baa76bdd3552cafab2182ac19598a3af0166ad43b472"
},
"evidence-047": {
"artifact": "full-pass-config.json",
"sha256": "3d64f88fff1153cd2d9a582d34463dac3fb24208dcd21d195a72fbb436a25fca"
},
"evidence-048": {
"artifact": "prepared-focus-filename-release-config.json",
"sha256": "6418fbd5b45ad12fda36733ba0849e41999637c3bf741bf070dcc73176a9de96"
},
"evidence-049": {
"artifact": "summary.json",
"sha256": "5ebc038fc3516ad955188b3c001c9938ca0494a77739995d3921de1b1958601a"
},
"evidence-050": {
"artifact": "protocol.json",
"sha256": "869a203487e775e021073010da24a01914360d470d168391d2ca8cfe259cd5bc"
},
"evidence-051": {
"artifact": "summary.json",
"sha256": "aafc7c95f10e9fb9fadfe4abca5f3ec3fe8c04b9ff7ecbc0c93c1f17dc5dfa24"
},
"evidence-052": {
"artifact": "summary.json",
"sha256": "c49db0eb4870539a8a96737e847994d41f07fdf7d30fe4ac7f047727ca896698"
},
"evidence-053": {
"artifact": "summary.json",
"sha256": "60feb0001979da6ce1de0a7a93a7e70028f7e4bfb32b2d56e5d4ef580c58e0e3"
},
"evidence-054": {
"artifact": "summary.json",
"sha256": "2bebb3439a722105cc00fbbdd479c6adb725bfa4c97d2c9e7d9c6ec4b6e00c47"
},
"evidence-055": {
"artifact": "summary.json",
"sha256": "15a910ba607c59da3ac2c57c4d31d648d41b9a3aec74b79a1beb56a6fed28765"
},
"evidence-056": {
"artifact": "summary.json",
"sha256": "a891bc9c6f170d8f682f410cd4a0eaa62f87879e33ed9da3b815ce85f487fe3d"
},
"evidence-057": {
"artifact": "summary.json",
"sha256": "fa3fd0454dcc24e9834e9df81abda5c9a3a8da81349d27cb8a1c1b7233db3206"
},
"evidence-058": {
"artifact": "summary.json",
"sha256": "78555df8bd904ede196fa684da6c5e57049310a450ac347a4d7fd7497c33c0a9"
},
"evidence-059": {
"artifact": "summary.json",
"sha256": "a18420af915f0e85101514e3cedc8e4d00f46aa71835abbf4067463de44b7fd2"
},
"evidence-060": {
"artifact": "summary.json",
"sha256": "96fa68183e2afdf896445f23af051973449348a825cedc28c6b3f5ce1efe37bf"
},
"evidence-061": {
"artifact": "summary.json",
"sha256": "6972048a836e5b870971e139bbf5425a7823c873171bbda930109fbf5e1260d5"
},
"evidence-062": {
"artifact": "summary.json",
"sha256": "71be99280c8ec627edb8ca3329779dc8ad780dd0080e548bcec781eb82abae82"
},
"evidence-063": {
"artifact": "summary.json",
"sha256": "5ce11eb42a439e6de3a89e3415f0d070f38f77afdc9c9457b7a379126373144c"
},
"evidence-064": {
"artifact": "summary.json",
"sha256": "4f06831e4758b3d4bcecc6190424c2279932f778d7a969d7dccbdd4f8483a8a2"
},
"evidence-065": {
"artifact": "summary.json",
"sha256": "ba06ac6025ea3a810435564c1b280736d33867b51f715e44ba0f2e114947428b"
},
"evidence-066": {
"artifact": "summary.json",
"sha256": "30de19eadad56d40a48a74787162e55fcda4825c0fa742bd3ded9c3e3fb1f5ce"
},
"evidence-067": {
"artifact": "summary.json",
"sha256": "e6088f0acd76f312987abc58f6676024a5d87b4ba335adec49ce5782e2ba882b"
},
"evidence-068": {
"artifact": "summary.json",
"sha256": "6a5b310143b91096d06510c40b6c1f4f749b70d0a6f282db10ce3050c7328a5d"
},
"evidence-069": {
"artifact": "summary.json",
"sha256": "0b4221d6df3007fa424adc15b558995197546351d9d8f62f07a20efaadef1ac6"
},
"evidence-070": {
"artifact": "summary.json",
"sha256": "367970958c57e340e5395d4f413e9891fcff4bad53a73345afbb86f0ea18eb4d"
},
"evidence-071": {
"artifact": "sent-development-comparison.json",
"sha256": "405b01f41be3300d5049aa51cf99f9544b32435768351a97865b77d4a401e209"
},
"evidence-072": {
"artifact": "summary.json",
"sha256": "d4705a0caf8e3753ec96a879e1f51afece67992025776a031ec3a0053cbb563f"
},
"evidence-073": {
"artifact": "summary.json",
"sha256": "6d37edfda256915bcf619758900065224b166c7de60f8d881c137f5fc210a7c1"
},
"evidence-074": {
"artifact": "summary.json",
"sha256": "2a292c05f3a3cae9ac539da67b496b119a4204556fa35c43d8c35d9f9d6b166e"
},
"evidence-075": {
"artifact": "summary.json",
"sha256": "7d43af1ded098d1fe1ec6ca8f1487cd4b7c308e5134d633efe0708b7e45cbbdf"
},
"evidence-076": {
"artifact": "aggregate.json",
"sha256": "c7c69da7d791c938930656d96e22995e5146663a0204ab2a65ff82407a3e9ab9"
},
"evidence-077": {
"artifact": "cases.json",
"sha256": "fe54ef3c62022d6dfbc9a35496d1217da00743f0eaf2ecec1dd875924ddd2aa3"
},
"evidence-078": {
"artifact": "predictions.jsonl",
"sha256": "54e4d8f134efc8bfdd55cf1c1801bc2d88e58621a04b5bf67b2e26c76317a9f1"
},
"evidence-079": {
"artifact": "results.json",
"sha256": "621904d6dc7eb868953600eb1ab9395707f57409002efea66f2dc8ab72362bd9"
},
"evidence-080": {
"artifact": "predictions.jsonl",
"sha256": "e70ec9fdabab9f02912cf378e1b02636c3341bb3ada3a953f0757a98a0e65595"
},
"evidence-081": {
"artifact": "results.json",
"sha256": "a31a353bf4a6267f4fe921005b0aad9a95cbb237df526c5002724a5fcc52add8"
},
"evidence-082": {
"artifact": "protocol.json",
"sha256": "50d6be667c3e4a2b6e3c800be44cddc48a0bf5d16938a736810aee92def2469e"
}
},
"training_fit": {
"gained": 105,
"lost": 0,
"preservation750_passed": 599,
"released_passed": 494,
"total": 600
}
}
References and suggested reading
For an accessible route through the concepts, begin with CheckList [7], then the text-to-SQL evaluation papers [5–6]. Read Self-Instruct [2] for synthetic instruction generation, followed by Learning without Forgetting [3] and InsCL [4] for preservation during adaptation. Dwork et al. [8] explain the statistical concern behind repeatedly reused evaluations; SynAE [9] connects directly to synthetic-versus-real agent evaluation.
- Binyuan Hui et al. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186.Why read it: background on the model family that supplies the starting checkpoint. Our narrow retrieval study does not reproduce its broad model evaluations.
- Yizhong Wang et al. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL, pp. 13484–13508.Reading question: what does filtering generated examples establish, and what still needs a separate downstream evaluation?
- Zhizhong Li and Derek Hoiem. 2016. Learning without Forgetting. ECCV; updated journal version available on arXiv.Reading question: how can a training objective encourage retention while adding a capability? The original setting is vision, so transfer to this language-model recipe should be reasoned about explicitly.
- Yifan Wang et al. 2024. InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions. NAACL.Reading question: how are replay examples chosen, and why might instruction similarity and training order matter? Our staged replay does not implement InsCL’s selection method.
- Catherine Finegan-Dollak et al. 2018. Improving Text-to-SQL Evaluation Methodology. ACL, pp. 351–360.Reading question: does a test require a new query structure, or mainly familiar structure with different wording and entity values?
- Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic Evaluation for Text-to-SQL with Distilled Test Suites. EMNLP, pp. 396–411.Reading question: how can multiple database instances distinguish queries that happen to agree on one instance? Relate this to the size-threshold fixture gap in Section 4.
- Marco Tulio Ribeiro et al. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. ACL, pp. 4902–4912.Reading question: which important capabilities are absent from a high-scoring benchmark? Build a coverage inventory before increasing the count of similar tests.
- Cynthia Dwork et al. 2015. Generalization in Adaptive Data Analysis and Holdout Reuse. arXiv:1506.02629.Reading question: how does observing earlier results affect the validity of later conclusions? This study records adaptive reuse; it does not implement the paper’s protected-holdout method.
- Shuaiqi Wang et al. 2026. SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations. arXiv:2605.22564.Reading question: which aspects of synthetic data resemble real use, and which do not? Its multi-axis framing helps separate validity from representativeness.