BENCHMARK · FINDBENCH

FindBench results: Shlexball leads on 312 unseen file search tests

26 September 2026 · updated 4 October 2026 at 11:47 am PDT (what changed)

Results

Every model received the same instructions Shlexball's app gives its own model. The comparison models also got 30 worked examples, and the board shows them with instructions only too. Shlexball's 1.5B model ran on a MacBook Pro (M4 Max). Gemini 3.8 Flash and Gemini 3.5 Flash-Lite ran on Google's servers, and Google does not disclose their sizes. Gemma 4 31B ran as the full uncompressed model on a rented NVIDIA H200 datacenter GPU.

Jeremiah v6 is the version name of Shlexball's model, fine-tuned from Alibaba's open Qwen2.5-Coder-1.5B-Instruct.

Updated 4 October 2026 at 11:47 am PDT. The main results now apply the plural step to every model, so all models are scored the way the app runs a search. Gemini 3.8 Flash was added to the board. No model was run again; we scored the answers each model had already given. Earlier the same day, at 11:37 am PDT, the plural step was first added as an extra column.

In the raw run, Shlexball produced 0 unsafe results and 0 wrong refusals.

Speed

Shlexball turned each request into a search in 0.41 seconds on average, faster than Gemini 3.5 Flash-Lite's 0.59 seconds — and Shlexball's number includes no network round trip, while Gemini's does.

How we measured: same 479 FindBench runs, on an M4 Max MacBook Pro (48 GB). Shlexball ran on the Mac itself, no network round trip; Gemini via Google's API, network included, 12 requests in flight. An “answer” means turning the request into a search, not searching your files. Gemma 4 31B is not in the speed comparison: we ran it on a datacenter GPU we rented and set up ourselves, with up to 64 requests at once, so its time would measure our setup, not a service you could use. Gemini 3.5 Flash-Lite is timed through Google's public API, which anyone can call. Slower Macs will be slower.

Every search starts with this step, so at well under half a second it never becomes the wait — the time you actually notice is the search over your files, which depends on how many you have.

How FindBench works

What a test looks like, how each answer is scored, what the 479 tests contain and how we checked that the 312 unseen tests never appeared in training are on the FindBench page.

Why a 1.5B model can win here

File search is a narrow job with a strict answer format and an exact right answer. With plurals, the app supplies singular and plural forms for every model, so the remaining gap comes from other conventions a model trained for this job learns: search for both “api_key” and “api-key”, and tell “a file that mentions a key” apart from “a file that stores a key”. A general-purpose model has to work these rules out from the instructions alone. Size helps when a task is broad. This task is not.

What the other models got

• Same instructions. The exact instructions Shlexball's app gives its own model (its “system prompt”), with temperature 0, which turns off deliberate randomness.
• A second run. Each comparison model also ran with 30 worked examples added.
• Generous clean-up. Before scoring, we fixed formatting slips: we removed markdown code fences, kept only the first line, and added the missing space after “docsearch”, the word every search plan starts with. Scored strictly, Flash-Lite with instructions only gets 36/479.
• Gemma 4 31B, run by us. We ran Google's full, uncompressed Gemma 4 31B ourselves, on an NVIDIA H200 datacenter GPU we rented, using the open-source vLLM server. Thinking was on, and each answer, thinking included, could use up to 12,000 tokens (word pieces); none was cut off. We first tried Google's own API, but it was too slow to finish: about a minute per answer, and some calls timed out. On the tests both runs covered, Google's API passed 56 of 181 and our run 51 (instructions only), and 68 of 140 against 67 (with examples); the two gave the same pass or fail on 168 of 181 and 121 of 140 tests.
• Thinking settings. “Thinking” is the step-by-step reasoning some models do before answering. Gemini 3.5 Flash-Lite ran at its default setting, minimal, and produced no thinking; each call allowed up to 2,048 output tokens, and its longest answer used 97. Gemma 4 31B's open release thinks only when asked, so we turned thinking on, matching Google's API, where it also thought on every answer. Gemini 3.8 Flash ran at its default, medium thinking, with up to 2,048 output tokens; that cap cut thinking short on about 20% of its answers. A partial re-run without the cap scored the same (46 against 47, and 94 against 94, on the tests it covered).

Limits

• FindBench is our own benchmark. The tests and the scorer are private for now.
• The 312 tests were never in training, but they helped us choose which Shlexball checkpoint to release, and the tool that wrote our training data also wrote them.
• In this scoring pass, the plural step saw only the folder each test expects. In the app it sees your real folders. Treat the plural numbers as a pilot.
• Gemini 3.8 Flash output was capped at 2,048 tokens, which cut its thinking short on about 20% of answers. A partial rerun without the cap scored the same.
• Google does not disclose the Gemini models' sizes.
• This benchmark covers one task, file search. It is not a general ranking of the models.
• Raw scores without the plural step, with 30 examples: Shlexball 280, Gemini 3.8 Flash 180, Gemini 3.5 Flash-Lite 151, Gemma 4 31B 145.

Benchmarks aside, the test that matters is your own files. Download Shlexball and try it: the first 30 searches are free.

▶ DOWNLOAD

macOS 15 or later. Apple silicon. 16 GB memory.