Tofazzol v1 and Findbench 3

Tofazzol v1 is the search model in Shlexball 1.12. It is a 1.5-billion-parameter model that runs on your Mac, and we trained it on a workstation we own, with no rented GPUs. On Findbench 3, our new development benchmark, it passes 703 of 746 file searches: documents, filenames, and photos, videos and audio by their metadata. Pristine, the model it replaced, passed 518.

We also ran three general models on the same file searches. Gemma 4 31B, an open model we ran on our own workstation, passes 733 of 746. Gemini 3.8 Flash passes 731 and Gemini 3.5 Flash-Lite 697. These general models received 30 worked examples with every request: sample requests paired with correct answers, often called few-shot examples. Tofazzol received none, and it is about twenty times smaller than Gemma.

Findbench 3 also scores Outlook mail plans on synthetic mailboxes, and Tofazzol passes 203 of 240 there. That number did not hold up in real use. On a real mailbox, Tofazzol reads "emails from september" as mail from a sender called September. The section Where the benchmark misled us explains why. The rest of this note covers the benchmark, every training run, what we fixed in the data, what we rejected, and what the comparison does and does not show.

Findbench 3: what it tests

Findbench 3 has 1,002 cases in 290 scenario clusters. Each case pairs a plain-English request with a small folder of files, or with a synthetic mailbox. That folder is built to contain the right answer and some near misses. The model writes one search plan. We run that plan and compare what comes back with the expected result.

A case passes only if the model's first answer returns exactly the expected files or messages, in order where order was asked for. We remove nothing but outer whitespace. There is no second try, and Shlexball does not repair the answer. Before any model was scored, we checked the benchmark itself. Every reference answer passed. A deliberately broad wrong answer failed for every case. And 3,454 plans with one constraint deleted (a missing sender, size, date or topic) were all rejected. That last check matters. A fixture is the test folder or mailbox a case runs against, and a fixture that holds only the obvious match cannot tell whether a plan quietly dropped half the request.

Findbench 3 is a development benchmark, and it has limits. We revised its distractors (the plausible wrong answers) once, after early predictions were in. Requests, contexts and answers stayed fixed. The 1,002 cases are variations within 290 clusters, not 1,002 independent tasks. The file or mail source is chosen for the model, so routing between them is not tested. The Outlook category tests plan writing on synthetic messages, not Shlexball reading a real mailbox.

The model and the recipe

Tofazzol starts from the published Qwen2.5-Coder-1.5B-Instruct checkpoint, not from Pristine. A checkpoint is a saved snapshot of the model's weights. All 1,543,714,304 parameters were trained: full supervised fine-tuning, with no adapter. The selected recipe has 23,976 training exposures over 23,433 distinct examples. These are 12,659 document searches, 4,048 filename searches, 3,220 email plans, 3,294 meaning (synonym) examples and 755 unsupported actions. The selected checkpoint saw 8,000 exposures, before the end of its first pass over the data.

Settings: AdamW with full-precision master weights and BF16 compute, loss on the answer only, batch 16, learning rate 2e-5 with 50 warm-up steps and a cosine schedule, weight decay 0.1 and gradient clipping at 1. Training ran on our Lenovo ThinkStation PGX (NVIDIA GB10, about 122 GiB of unified memory). It ran at about six seconds per step and 96% GPU utilisation, and the selected run trained for about an hour. We logged training loss as a diagnostic. Loss never chose a checkpoint; executed tests did.

Every run

We scored all ten Tofazzol checkpoints completely, each on the same frozen board. Two things stand out. The best total is not the last checkpoint, and no run improved every category.

Against Pristine, Tofazzol gains 409 cases and loses 21. Against the best earlier composition checkpoint it gains 145 and loses 43: documents +63/−1, filenames +1/−6, media +3/−14 and mail +78/−22. Training past step 500 hurt. Step 616 gained 37 cases and lost 125. We stopped because the target was met, not because we had shown a plateau, and we ran one training seed (one random starting point).

The biggest single change was between the broader audited recipe and the mail-cleaned one: 751 to 906. The only intended difference was removing defective mail examples. But removing rows also changes the batch order, so we treat the size of that jump as encouraging rather than measured.

What we fixed in the data

Most of the work was in the training data, not the optimiser (the part of training that updates the weights). The defects below were found by executing examples against fixtures written to catch them. None of them suggests that any author, human or model, wrote bad examples on purpose.

• Inherited label defects. 75 rows were excluded on structural and policy grounds, and 17 more after counterexamples showed their targets were wrong. Of 152 document rows flagged for review, 67 were proven defective. The other 85 were kept: a flag is not proof.
• Mail phrase recall. 526 inherited mail targets required a topic to appear as one contiguous phrase, so they missed ordinary paraphrases. 146 hand-written alternative wordings and 1,060 omission controls showed it. The original fixtures had repeated the exact phrase, so these targets had looked correct. Removing them is the mail-cleaned recipe.
• Language review. All 512 new mail drafts were read, and 24 were replaced. An early automatic check had counted the word "mail" inside email addresses. The corrected check fixed that.

The lesson we keep relearning: re-running an example's own fixture proves very little. A fixture has to contain the near miss, whether that is the other sender, the older file or the same words split across a sentence. Otherwise a wrong target passes.

What we tried and rejected

• A prompt primer ahead of each request. It scored 140 of 986, against 514 for the same model without it.
• Turning off the repetition penalty. One checkpoint fell from 606 to 550, and Pristine fell from 518 to 502. Applying the penalty to output tokens only also lowered a diagnostic board, from 415 to 392 of 512.
• More epochs (passes over the training data), more memory or lower loss. None of them produced better executed search on its own.

Two diagnostics point at what remains. Replacing only the topic words in Tofazzol's failed document plans with the reference topics fixes six of its seven document failures. Replacing the non-topic mail fields fixes all 37 mail failures, 25 of them from the sender or recipient alone. The remaining errors are mostly about copying names and binding them to the right field, not about vocabulary. These substitutions are diagnostics, not scores.

How general models do on the same board

We gave Gemini 3.8 Flash, Gemini 3.5 Flash-Lite (the newest text Flash-Lite in Google's catalogue on 6 November) and Gemma 4 31B the same instructions Tofazzol uses, plus 30 worked examples for each source. That meant 30 for file searches and 30 for mail, written for this comparison with made-up names, folders and topics. Before we froze the examples, we executed each one on its own fixture with near misses, and 60 of 60 passed. None repeats a benchmark request. Each model saw only the instructions, those examples and the request. It never saw a fixture, an expected answer or another model's output. We used temperature 0 and one answer per request, with medium thinking for both Gemini models and thinking on for Gemma, and we scored every answer with the same executor. Tofazzol and Pristine used their own trained instructions and no examples.

By category, Gemini 3.8 Flash passes 319 of 320 document searches, 158 of 160 filename searches, 254 of 266 media searches and all 240 mail plans. Against Tofazzol it gains 69 cases and loses 4; 37 of its gains are mail plans and 16 are media. Flash-Lite gains 65 and loses 41. It is behind Tofazzol on filenames (141 against 150) and documents (310 against 313), and ahead on media (246 against 240) and mail (233 against 203). Gemma 4 31B passes 320 of 320 document searches, 158 of 160 filename searches, 255 of 266 media searches and 240 of 240 mail plans; against Tofazzol it gains 69 cases and loses 2. No Gemini or Gemma answer was blocked, empty or cut off. The two Gemini runs cost about 4.10 US dollars at list prices.

What this does and does not show. Gemma 4 31B and Gemini 3.8 Flash write better search plans for this board than Tofazzol does, by a clear margin, and we report that plainly. Gemma's result is the more interesting one for us: it is an open model, and it ran on our own hardware. But at 31 billion parameters its weights alone take about 58 GB of memory, far more than the 16 GB Macs Shlexball supports. The comparison is between complete recipes, not identical prompts. The general models received 30 worked examples that cover the request shapes this benchmark uses, and Findbench 3 is a templated development board where that kind of coverage helps a great deal. Tofazzol runs on your Mac: your files are never uploaded, and there is no per-search cost. The Gemini rows send every request to Google. The result tells us where the ceiling is for plan writing on this board. It also tells us that our next gains should come from field binding and name copying, which is where Tofazzol loses most of its remaining cases.

Where the benchmark misled us

After Shlexball 1.12 shipped, we tried Outlook search on our own Macs. It was weak. So we ran Tofazzol on ten ordinary requests and ran its plans against one Mac's real Outlook store. We looked only at counts, never at the mail itself.

Folders, topics and "everything" work. Dates do not. There are two reasons, and the benchmark tests neither.

• Months and years. The mail plan format can only say "the last N days". It has no way to say "in September" or "in 2025". No training example used a month or a year either. And every Findbench 3 mail request that says "from" is followed by a person. So the model learned that "from" always means a sender.
• An out-of-date copy. Shlexball searches the copy of your mail that Outlook keeps on the Mac. On this Mac, Outlook last updated that copy on 25 September, so "last week" finds nothing. Shlexball shows that date with every answer, but it is easy to miss.

The lesson: a benchmark built from the same patterns as the training data will reward those patterns. Findbench 3's mail requests were tidy ("emails from Hester Sutton about clinic sanitation"), and its synthetic messages carried their full text, while Shlexball sees only the short preview Outlook stores. The 203 of 240 measured how well Tofazzol fills in a tidy template, not how well Shlexball finds your email. The file searches stand on firmer ground, because they run through Shlexball's real search engine on real files. But they are templated too: a bare request such as "adpig demo" missed a video that "adpig demo video" found. Next we will add requests written the way people actually type them, months and years, and out-of-date mail copies to both the training data and the benchmark, and test on real Macs before we quote a number.

What shipped

Shlexball 1.12.0 (6 November) ships Tofazzol v1 and a first version of Outlook mail search on the Mac. Inside Shlexball, Tofazzol's file result rises from 703 to 709 of 746, because Shlexball's own photo, video and audio rules fix six more cases and break none. The release notes also state what got worse. On Findbench 2.0, the earlier board, Tofazzol passes 349 of 733 against Pristine's 580, and 28 of 89 confirmation requests against 61. These misses are mostly searches for saved keys and passwords, provider names and topic variants. Shlexball 1.12.1 followed the same day. The 3.1 GB model download had been pausing itself on slower connections, and the download server was slower than it should have been. Both are fixed.

Limits and next steps

• Outlook: email search is early. It handles folders and topics but not months or years (see above). On a work Mac the first test found nothing while the new model was still downloading; we will test it again. Recipients, attachment names and full message bodies are not read yet, and searches that need them say so.
• Media names: a request such as "adpig demo" missed a video that "adpig demo video" found. Documents do not have this problem. We will add paired examples, with and without the kind word, to the next training data and to the benchmark.
• Findbench 3 is a development board and we ran one training seed. An untouched confirmation set and a second seed come before any stronger claim.
• Next: training aimed at copying names and binding them to the right field, the regressions on saved credentials and provider names, and a confirmation run with Gemini and Gemma on the same untouched set.