Tofazzol v2 and Findbench 3.1

Tofazzol v2 is the search model in Shlexball 1.13, released on 7 November. It is the same size as Tofazzol v1, 1.5 billion parameters, and it runs on your Mac. We trained it further from v1 on our own workstation, with more email examples in its training: sender names copied exactly from the request, and dates such as a month or a year. On the 746 file searches of Findbench 3.1 it passes 735, up from 703. That is more than Gemma 4 31B (733) and Gemini 3.8 Flash (731), which each received 30 worked examples with every request. Tofazzol received none.

Findbench 3.1 adds 240 email searches to those file searches, for 986 in all. Tofazzol v2 passes 952 of 986. Gemma 4 31B (973) and Gemini 3.8 Flash (971) are still ahead overall: they pass all 240 email searches, and Tofazzol v2 passes 217. On 30 real emails in a work mailbox, Tofazzol v2 found the wanted email every time, and 27 of its answers were exact.

Tofazzol v2 on Findbench 3.1

In Findbench 3.1, a test passes only if the model's first answer, actually run, returns exactly the expected files or emails. Tofazzol v2, a 1.5B model on your Mac, leads file searches and media, stays close on documents and filenames, and trails on email.

Tofazzol v2 · 1.5B · your Mac · 0 examplesOther models · 30 examples per request

File searches

First at 98.5 percent; the runner-up has about twenty times the parameters.

98.5%

98.3%

98.0%

93.4%

94.2%

Tofazzol v2Gemma 4 31BGemini 3.8 FlashGemini 3.5 Flash-LiteTofazzol v1

Media

First again at 98.5 percent, seven searches ahead of Gemma 4 31B.

98.5%

95.9%

95.5%

92.5%

90.2%

Tofazzol v2Gemma 4 31BGemini 3.8 FlashGemini 3.5 Flash-LiteTofazzol v1

Documents

Gemma leads at 100 percent; Tofazzol v2 trails by two searches, at 99.4.

99.4%

100%

99.7%

96.9%

97.8%

Tofazzol v2Gemma 4 31BGemini 3.8 FlashGemini 3.5 Flash-LiteTofazzol v1

Filenames

Tofazzol v2 reaches 96.9 percent, three searches behind the 98.8 percent leaders.

96.9%

98.8%

98.8%

88.1%

93.8%

Tofazzol v2Gemma 4 31BGemini 3.8 FlashGemini 3.5 Flash-LiteTofazzol v1

Email

Gemma and Gemini reach 100 percent; Tofazzol v2 trails at 90.4.

90.4%

100%

100%

97.1%

84.6%

Tofazzol v2Gemma 4 31BGemini 3.8 FlashGemini 3.5 Flash-LiteTofazzol v1

Model size

Tofazzol v2: 1.5 billion. Gemma 4 31B: 31 billion. Gemini: not published.

1.5B

31B

not published

not published

1.5B

Tofazzol v2Gemma 4 31BGemini 3.8 FlashGemini 3.5 Flash-LiteTofazzol v1

Tofazzol v2's 262 media searches are the most on the board; Gemma passes 255. Against v1, it gains 33 file searches and loses 1, a filename search. On email it gains 16 and loses 2. Inside Shlexball, its file result is 736 of 746, because Shlexball's own photo, video and audio rules fix one more case. The table's scores are for the model as Shlexball 1.13 ships it, measured through the release's own model package. Before the release, the same version of the model scored 216 of 240 on email (951 in total); the release run passed one more email search.

Findbench 3.1: what it tests

Findbench 3.1 uses the cases, folders, mailboxes and scoring of Findbench 3, which we introduced on 6 November. The only change is how we report them: file and email searches counted together, out of 986. On 6 November we left email out of the headline, because Tofazzol v1's email score did not hold up in real use. Tofazzol v2 does much better on real email (see Real email), so we count email again.

The benchmark has 1,002 cases in 290 scenario clusters. Each case pairs a plain-English request with a small folder of files, or with a synthetic mailbox. That folder or mailbox is built to contain the right answer and some near misses. The model writes one search plan. We run that plan and compare what comes back with the expected result.

A case passes only if the model's first answer returns exactly the expected files or emails, in order where order was asked for. There is no second try, and Shlexball does not repair the answer. Before any model was scored, we checked the benchmark itself: every reference answer passed, a deliberately broad wrong answer failed every case, and 3,454 plans with one constraint deleted (a missing sender, size, date or topic) all failed. That last check matters, because a test folder that holds only the obvious match cannot tell whether a plan quietly dropped half the request.

Findbench 3.1 is a development benchmark, and it has limits. We revised its plausible wrong answers once, after early predictions were in; requests and expected answers stayed fixed. The 1,002 cases are variations within 290 clusters, not 1,002 independent tasks. Whether a request goes to files or to email is decided for the model, so that choice is not tested. And the email category tests search plans on synthetic messages, not Shlexball reading a real mailbox, which is why we also tested real email.

How the general models were run

On 6 November we gave Gemini 3.8 Flash, Gemini 3.5 Flash-Lite and Gemma 4 31B the same instructions Tofazzol v1 uses, plus 30 worked examples per request, matched to its source: 30 for file searches and 30 for email, written for this comparison with made-up names, folders and topics. Each example passed on its own test folder before we froze them, and none repeats a benchmark request. Each model saw only the instructions, those examples and the request, never a test folder, an expected answer or another model's output. We used temperature 0 and one answer per request, with medium thinking for both Gemini models and thinking on for Gemma, and ran and scored every answer the same way. Tofazzol v1 and v2 used the instructions they were trained with and no examples; v2's email instructions also cover calendar dates.

Gemma 4 31B and Gemini 3.8 Flash write better email plans for this benchmark than Tofazzol v2 does, and we report that plainly. Gemma is an open model and ran on our own workstation, but at 31 billion parameters its weights alone take about 58 GB of memory, far more than the 16 GB Macs Shlexball supports. The Gemini rows send every request to Google. Tofazzol runs on your Mac: your files and email are never uploaded, and there is no per-search cost. The general models' 30 worked examples cover the request shapes this templated benchmark uses, which helps them a great deal here.

Real email, not only the benchmark

A benchmark score is not the same as finding your email, so we also tested Tofazzol v2 on a real work mailbox. We picked 30 emails in Outlook on one of our Macs and wrote one request for each, the way a person would ask: by subject, by sender and subject, by date and subject, or by words in the email and its sender. We fixed the requests and the expected answers before any model answered. Then we ran each model's first answer through Shlexball's own mail reader. The mail and the requests stayed on that Mac; only the counts below left it.

Tofazzol v2 found the wanted email in all 30 searches, and 27 answers were exact: the wanted email found and every other email ruled out. The other three asked for words inside an email. Outlook keeps the full text of only some emails on the Mac, so Shlexball could not rule out other matches, and it said so instead of claiming it had checked everything. All five requests that named a date passed: Tofazzol v2 can search by calendar month, year or named period ("in September", "in 2025"), which v1's search format could not express. Part of this improvement is Shlexball rather than the model. Before a fix in Shlexball 1.13, it could not read the folders Outlook keeps for this kind of work account, and every model found 0 of the 30.

One limit remains from 6 November: Shlexball searches the copy of your mail that Outlook keeps on the Mac. If Outlook has not updated that copy lately, newer mail is not there, and Shlexball shows when Outlook last updated it.

How we chose Tofazzol v2, and what that means

Tofazzol v2 is one of 64 versions we scored on this benchmark on 6 and 7 November, checkpoints saved during training and blends of them, and we used the 30 real emails to choose among the last four. So both results are development results, not independent tests. A model picked from many on one benchmark can fit that benchmark, and these 30 emails helped pick it. Another candidate scored higher on Findbench 3.1, 969 of 986 with all 240 email searches. But it passed fewer file searches (729) and fewer real emails exactly (26 of 30). We chose the candidate that did best on files and on real email. An untouched confirmation set still comes before any stronger claim.

What got worse

Tofazzol v2 fails one filename search that v1 passed. On our separate photo, video and audio board, one photo search got worse ("images downloaded from chrome" now also asks for a folder named chrome), so it passes 59 of 64 checks against v1's 60. On Findbench 2.0, our older benchmark, it passes 450 of 733 against v1's 349, and 36 of 89 confirmation requests against v1's 28.

What shipped

Shlexball 1.13.0 (7 November) ships Tofazzol v2. It also makes email part of every search: a request that says email, mail, inbox or Outlook searches the mail New Outlook keeps on the Mac, with no setting to turn on. It reads the folder layout newer versions of Outlook use for work and school accounts, and when Outlook keeps an email's full text, word searches read it, not only the preview.

Limits and next steps

• Email: recipients and attachment names are not read yet, and full text only where Outlook keeps it. Searches that need them say so.
• Findbench 3.1 is a development benchmark, and we chose Tofazzol v2 on it and on the 30 real emails. An untouched confirmation set, run with Gemini and Gemma too, comes before a stronger claim.
• Next: the 23 email searches Tofazzol v2 still misses on this benchmark, and the older Findbench 2.0 requests.