An earlier benchmark. Findbench 3, published 6 November 2026, is now Shlexball's main benchmark. This page keeps the Findbench 2.0 definition and its October results. How Findbench 3 works
SHLEXBALL RESEARCH · 733 EXECUTED TESTS
findbench 2.0
733 file-search tests under one defined search contract. A model turns a request into a search plan or refusal. We execute the answer on a controlled test folder and require exactly the expected files, or exactly the expected refusal code.
The October unaided reference result is Pristine_Repair_Unaided: 580/733 tests passed (79.1%). This is an experimental model and a development benchmark. Independent confirmation is measured separately.
Defined 13 October 2026 · Internal results · Scoring contract · Earlier FindBench methodology
The fixed 733-case board
findbench 2.0 names a frozen, contract-aligned evaluation: 657 independently reviewed cases from an earlier 703-case development board, plus 76 newer PDF and human-style cases. The review corrected file-type expectations in 211 cases. Another 46 historical cases were excluded before this version was frozen: 26 with unresolved semantic or representation issues, and 20 without usable independent review.
Each case counts once. The score is tests passed divided by 733; family percentages are not averaged. The 76 newer cases are already included in 733 and must not be added again.
| Suite | Cases | What it tests |
|---|---|---|
| Retained search skills | 447 | Topics, credential distinctions, provider names, scope, plurals, alternatives and limits |
| PDF and human-style requests | 76 | Concepts, paraphrases, composition, fragments, typos, scope and time |
| Filename searches | 53 | Names, extensions, folders, size and file age |
| PowerPoint searches | 53 | Content, format, identity and composed constraints |
| Search/refusal distinctions | 54 | Supported requests versus unavailable capabilities, valid fields and refusal codes |
| Format selection | 50 | Requested file formats versus words that belong to the document topic |
| Total | 733 | One frozen board for every reported model |
The search and scoring contract
- Preserve the request. A plan must keep its topics, Boolean constraints, exclusions, explicit format, scope, time, identity, order and limits. A related concept is not automatically an equivalent meaning.
- Distinguish discussion from stored credentials. Ordinary document-topic searches do not implicitly include
.envconfiguration files. Unrestricted searches for actual stored credentials include them. An explicit request for.envis respected. - Execute the answer. The native search engine runs it on a controlled fixture. A search passes only with the exact expected paths, applicable order and limits, complete execution and no error. A partial match or a plausible-looking plan does not pass.
- Require the correct refusal. An unsupported request passes only with its exact expected refusal code. Refusing a supported search is a failure.
- Keep safety enforced. Attempts to write, escape the allowed folder or reveal credential values fail. Native validation, permissions and read-only protections remain active.
Scoring is based on executed results, not token loss or a model judging a candidate answer. Test intent and expected results were reviewed before the board was frozen. Unresolved requests that the current execution contract cannot represent remain outside this version; they are not silently reinterpreted to make a model pass.
Unaided and assisted runs are labeled separately
Unaided: one model answer is sent directly to the native executor. There is no app semantic rewrite, synonym-helper question, format or folder correction, refusal recovery, or empty-result retry. The executor still supplies file readers, parsing, ranking and safety checks.
Existing app assistance: the app may normalize or replace a plan, request meanings or a follow-up answer, and retry an empty search. Every model supplies its own follow-up answers. This measures the complete model, prompt and app recipe, and is labeled explicitly.
Future model comparisons will use the same frozen cases and exact scoring. We will disclose the model version, prompt and worked examples, decoding settings, assistance mode and call budget. Scores under different assistance modes are not presented as a weights-only comparison.
Internal comparisons, October 2026
These runs use the same 733 cases and pinned execution contract. Each model keeps its trained prompt. Other models from the earlier FindBench comparison have not yet been evaluated on findbench 2.0.
| Model | Assistance | Passed | Percentage |
|---|---|---|---|
| Pristine_Repair_Unaided | None | 580/733 | 79.1% |
| v6 | Existing app assistance | 481/733 | 65.6% |
| v6.1 | Existing app assistance | 466/733 | 63.6% |
| v6 | None | 338/733 | 46.1% |
| v6.1 | None | 398/733 | 54.3% |
The assisted comparisons start from each model's frozen first answer, then use that same model for any follow-up. The saved model-owned turns were replayed through fresh native execution to verify the comparison. Pristine's unaided result is the benchmark to beat; its weights and prompt are preserved under that name.
Pristine remains uneven: it passes 405/447 retained-skill tests, 50/53 filename tests, 38/50 format tests, 29/53 PowerPoint tests, 29/54 search/refusal tests and 29/76 PDF/human-style tests. PDF concept composition is still 0/11. The overall percentage must not hide those weaknesses.
Development and independent confirmation
The 733 cases have informed development. We do not claim that all 733 are unseen training examples or an untouched holdout. Repeated progress on this board must be checked against separately authored, reserved confirmation cases.
The completed training pilot's separate 89-case confirmation set produced 61/89 (68.5%) for Pristine unaided: 43/71 supported searches and 18/18 correct refusals. Those results have now informed diagnosis, so this set becomes a regression check. The next training round needs a new frozen confirmation set, excluded from training and opened only after candidate selection.
The training target is to approach 90% on both the contract-aligned board and independent confirmation, while retaining existing passing cases. It is a goal, not a result or a guarantee. On this fixed board, reaching 90% requires at least 660/733 passes.
Version identity and limitations
This public name refers to the exact existing freeze; naming it does not change any case or result. Any future change to the cases, expected outcomes or execution contract requires a separately identified version and new comparable runs.
- Scoring version
shlex-pristine-contract-aligned-development-v1- 733-case freeze SHA-256
94beac18aeaca08db13a3d3f322095d8b6e759e8af0b5ff64d2201d378828589- Native executor SHA-256
60c4b33fe880bb4cbe22ae9c90bbd34629614573470ae6ec20d9d9e203908478
This is Shlexball's own benchmark. The cases, fixtures and complete scoring package remain private; hashes identify the frozen artifacts but do not make the results independently reproducible on their own. The suite tests a defined file-search contract, not unrestricted Bash programming or every way a person might search.
The earlier 666/703 incumbent result uses different labels and scoring. It is not a findbench 2.0 score. The earlier public 479-case board and its 312-case subset are also separate: their results and wording remain available in the methodology archive and historical results article.