Tofazzol v2.1: 80 of 100 held-back email requests

10 November 2026 · 1,962 words · 9 minute read

Tofazzol v2.1 is the search model in Shlexball 1.16, released on 10 November. It is the same size as Tofazzol v2, 1.5 billion parameters, and it runs on your Mac. It is a short further training of v2: the same data and the same method, 600 more steps at a low learning rate. We made it to answer Microsoft Outlook email requests better, and we held back a set of requests to find out whether it does.

On those 100 held-back requests, Tofazzol v2.1 returned exactly the right emails for 80. Tofazzol v2 returned them for 66. The email that was wanted was among the results for 90, up from 77. On Findbench 3.1 the two models are level, at 952 of 986.

Email search on a real mailbox

100 held-back requests about one real work mailbox (n=100)

80%Tofazzol v2.1
66%Tofazzol v2

Higher is better. A request counts only when the model's first answer returns exactly the expected emails and Outlook's copy on the Mac holds enough to confirm it.

How the 100 requests were held back

The requests are about a frozen copy of one real work mailbox, as Outlook keeps it on a Mac. A script wrote them: ten kinds of request, ten of each, every one aimed at a different email. None of those emails had been a target in the 130 requests we had already used. We froze the set before any of the new training runs produced a model, and we did not look at it while choosing one.

We chose the model with other tests: a development set of 100 requests from the same mailbox, the 30 real work emails from the Tofazzol v2 note, and Findbench 3.1. Ten training runs gave 72 saved models. The rule for picking one was written down first: among the models that lost none of the 30 real emails and kept every calendar and older-mailbox check, take the one that lost the fewest tests it had passed before. That was the model we now call Tofazzol v2.1. Only then did we score the held-back 100, once.

A request passes only if the model's first answer, run by Shlexball's own mail search, returns exactly the emails a reference search returns, and the reference search itself could see everything it needed. Nine of the ten topic requests fail that last condition for every model, because Outlook does not keep the full text of those messages on the Mac. They stay in the 100, so the most any model can score is 91.

Where the gain comes from

The same 100 held-back requests by kind, ten of each · requests answered exactly, out of 10

Tofazzol v2.1
Tofazzol v2

A subject and a month90
A folder and a subject95
A sender and a subject97
A subject1010
Mail you sent, by subject1010
A sender's address99
Replies and forwards you sent99
A sender's name77
A sender and a month89
A topic in the message text9 of 10 cannot be confirmed for any model00

Longer is better. Tofazzol v2.1 gains 15 requests and loses 1. The loss is a request for a sender and a month, which it answered with a search that cannot run.

What changed in its answers

Most of the gain is one kind of request: a subject and a month, such as an email about a named project from a given month. Tofazzol v2 read the word before the month as the start of a sender's name and searched for a sender who does not exist. It did that in all ten, and found nothing. Tofazzol v2.1 treats the month as a date in nine of the ten.

The same habit shows across the whole set. Tofazzol v2 put a sender into 59 of its 100 searches and a date into 10. Tofazzol v2.1 put a sender into 51 and a date into 19. Requests that name a folder and a subject went from 5 to 9, and requests that name a sender and a subject from 7 to 9.

Three kinds did not move: a sender's name alone stays at 7 of 10, a sender's address at 9 and replies and forwards you sent at 9.

Findbench 3.1: level

Findbench 3.1 is 746 file searches and 240 email searches on synthetic mailboxes, described in the Tofazzol v2 note. Tofazzol v2.1 passes 952, the same as v2. They are not the same 952: v2.1 passes three searches v2 fails and fails three that v2 passes.

File search

Findbench 3.1 file searches (n=746)

98.5%Tofazzol v2.1
98.5%Tofazzol v2
94.2%Tofazzol v1
98.3%Gemma 4 31B
98.0%Gemini 3.8 Flash
93.4%Gemini 3.5 Flash-Lite

Synthetic email search

Findbench 3.1 email searches on synthetic mailboxes (n=240)

90.4%Tofazzol v2.1
90.4%Tofazzol v2
84.6%Tofazzol v1
100%Gemma 4 31B
100%Gemini 3.8 Flash
97.1%Gemini 3.5 Flash-Lite

Higher is better. Every model's first answer runs through the same search engine, and a search passes only if it returns exactly the expected files or emails. Tofazzol answers from its training alone, on your Mac. The other models received the same instructions plus 30 worked examples with every request.

Every test, side by side

Share of requests answered exactly

Test	Tofazzol v2.1Shlexball 1.16 · on your Mac	Tofazzol v2Shlexball 1.13 to 1.15	Tofazzol v1Shlexball 1.12	Gemma 4 31BOur workstation · 30 examples	Gemini 3.8 FlashCloud · 30 examples	Gemini 3.5 Flash-LiteCloud · 30 examples
Email on a real mailboxHeld-back requests · exactly the right emails	80%	66%	Not run	Not run	Not run	Not run
Email on a real mailboxHeld-back requests · wanted email among the results	90%	77%	Not run	Not run	Not run	Not run
Email on a real mailboxDevelopment requests · exactly the right emails	83%	72%	Not run	Not run	Not run	Not run
File searchFindbench 3.1 · 746 searches	98.5%735 of 746	98.5%735 of 746	94.2%703 of 746	98.3%733 of 746	98.0%731 of 746	93.4%697 of 746
Documents320 searches	99.1%317 of 320	99.4%318 of 320	97.8%313 of 320	100%320 of 320	99.7%319 of 320	96.9%310 of 320
File names160 searches	97.5%156 of 160	96.9%155 of 160	93.8%150 of 160	98.8%158 of 160	98.8%158 of 160	88.1%141 of 160
Photos, video and audio266 searches	98.5%262 of 266	98.5%262 of 266	90.2%240 of 266	95.9%255 of 266	95.5%254 of 266	92.5%246 of 266
Email on synthetic mailboxesFindbench 3.1 · 240 searches	90.4%217 of 240	90.4%217 of 240	84.6%203 of 240	100%240 of 240	100%240 of 240	97.1%233 of 240
All of Findbench 3.1986 searches	96.6%952 of 986	96.6%952 of 986	91.9%906 of 986	98.7%973 of 986	98.5%971 of 986	94.3%930 of 986

Bold marks the best score in each row. A dash means the model was not run on that test: the real-mailbox requests were run only on Tofazzol, on the Mac that holds the mailbox. Tofazzol answers from its training alone; Gemma and Gemini were given the same instructions plus 30 worked examples with every request. A search passes only when the model's first answer returns exactly the expected files or emails.

These are first answers run directly by the search engine. Inside the app, a few of Tofazzol's file answers are corrected before they run. Counted that way, v2.1 passes 735 of the 746 file searches and v2 passed 736.

Every check before release

We ran both models through the release build of Shlexball 1.16 in one sitting. The first two rows are the held-back requests. Everything else was also used, at some point, to choose or to check the model.

Tests passing · Tofazzol v2.1 against Tofazzol v2
Check	Tofazzol v2.1	Tofazzol v2	Out of
Held-back real-mail requests, exact	80	66	100
Held-back real-mail requests, wanted email found	90	77	100
Development real-mail requests, exact	83	72	100
Original 30 real work emails, exact	27	27	30
Original 30 real work emails, wanted email found	30	30	30
Findbench 3.1 file searches, through the app	735	736	746
Findbench 3.1 email searches	217	217	240
New-email development set (v2 with the app's Sent-folder rule)	197	180	200
Calendar dates on a real mailbox layout	147	147	147
Media board, strict checks, through the app	59	59	64
Photo and video label fixture	7	7	8
Findbench 2.0	452	450	733
Older confirmation board	34	36	89

Two rows go the other way. Through the app, v2.1 passes one Findbench 3.1 file search fewer than v2. On an older confirmation board of 89 file searches it passes 34 where v2 passed 36. Neither model returned anything unsafe.

We also checked that Shlexball's own rules do not work against the new model. The app adjusted 50 of its 746 file answers and turned none of them from right to wrong. The rule that stops a search for someone's email from looking only in Sent, added in 1.15, fired on 2 of its 259 real-mailbox answers, and both were already wrong.

What this does not show

It is one mailbox. Another mailbox, with other senders, folders and habits, may give a different answer.
A script wrote the requests, in the same ten patterns as the development set. People type differently. Expect a smaller gain on your own mail.
One hundred requests is a small test. Fifteen gained and one lost is a clear direction, not a precise size.
Email searches that depend on who a message was sent to still depend on what Outlook has downloaded to the Mac. This model does not change that. On a copy Outlook has just rebuilt, such a search confirms nothing, with either model.
After release we replayed 46 ways of asking for email you sent through both models. They wrote the same search for 40. Tofazzol v2.1 was better on three, all about replies and forwards, and worse on three: asked for the last email to a person it sometimes asks for the oldest, and once it named a folder that does not exist.
On synthetic email searches, Gemma 4 31B and Gemini 3.8 Flash still pass all 240, and Tofazzol v2.1 passes 217.

What we learned making it

The useful gain came early in training, and a longer run lost it again. We nearly missed it, because the earlier run had saved too few models along the way.
The model that scored highest on the development set was not safe to ship: it lost 4 of the 30 real work emails. Averages of several models failed a check on an older mailbox layout. We kept the rule of counting what each model lost, not only its total.
Whether a run fixed searches that wrongly looked only in Sent depended on its random seed. Extra training examples fixed that on every seed, and each time cost one or two of the real work emails. We put the fix in the app instead, as a plain rule, in Shlexball 1.15.

Earlier notes: Tofazzol v2 and Findbench 3.1, and the working papers Small Models, Narrow Tasks and Synthetic Email Gains and Executed Retrieval.