Small Models, Narrow Tasks

1. The question: how much model does a search interface need?

A general language model must handle an enormous variety of requests. A desktop search planner has a more specific responsibility: interpret a request, copy the relevant names or phrases, and express the right constraints in a format a search engine understands. The executor then reads the files or mailbox, applies the filters and returns the results.

This division of work creates an opportunity for a small model. A useful search planner does not need to reproduce every capability of a large conversational model. It needs to perform the transformations required by its interface reliably. Our research question is whether extensive specialization can make a 1.5B model competitive with strong general models on that interface.

The scope matters. File search includes document contents, filenames and media metadata; email is reported separately. The experiment measures whether a model’s plan retrieves the expected results from a fixture. It does not measure general reasoning, conversation, vision, open-ended research or the full experience of searching an arbitrary personal computer.

This paper makes three contributions: a comparison of saved answers under one execution setup; a paired analysis showing the gains and losses behind similar totals; and an explanation of the specialization mechanisms that the system uses, with a clear separation between observed design and untested causation. The result is an empirical case study of an established phenomenon, rather than a claim to have discovered small-model specialization.

2. What the model learns, and what the software does

Tofazzol begins with Qwen2.5-Coder-1.5B-Instruct, a pretrained code-oriented language model. The underlying model family is described by Hui et al. [1] The production weight file contains 1,543,714,304 parameters in BF16 storage. Its language and code knowledge therefore precedes the project’s task-specific training; the project did not train a language model from scratch.

Search examples teach the model how to turn language into the product’s output contract. The model must distinguish a sender from a subject phrase, select file or mail constraints, preserve literal text and choose appropriate dates or ordering. Its output is a single plan. The corpus to be searched is supplied to the executor, rather than inserted into every model prompt.

The executor handles exact operations that would be wasteful to reproduce inside a model’s weights: traversing directories, reading supported formats, normalizing text and evaluating filters. A large comparator receives the same opportunity to use this executor. The observed difference is in plan generation, under each model’s adaptation recipe.

A compact output contract also limits the kinds of programs the model needs to produce. That does not guarantee correct interpretation. A syntactically valid plan can copy the wrong name or attach a date to a sender field. The distinction between syntax and meaning will matter in the results and limitations.

3. Models, adaptation and measurement

3.1 Compared recipes

The exact generalist identifiers were gemma-4-31b-it, gemini-3.8-flash and gemini-3.5-flash-lite. The Gemini response receipts return those respective Gemini names. Gemma used the recorded google/gemma-4-31B-it deployment. Official model descriptions provide the size and family context. [2] [3] [4] We assign no parameter ratio to the Gemini models. Only the listed Flash-Lite version was evaluated.

The original external-model runs were recorded on 6 November 2026. All used temperature 0 and one answer per request. Gemini allowed up to 65,536 output tokens; Gemma allowed 12,000, including thinking. The recorded Gemma server was vLLM 0.33.0 with a 16,384-token context limit. All 1,002 historical responses per generalist ended normally; the recorded counts for blocked, empty, truncated and failed answers were zero. These are specific tested configurations, not a search for each model’s best possible configuration.

The generalists received 30 demonstrations for file requests and 30 for mail requests, drawn from 60 authored examples. All 60 reference examples passed execution on their own fixtures. The overlap audit found no exact request duplication with the board. However, the demonstrations cover familiar task structures and vocabulary; absence of exact duplicates does not establish structural independence. Tofazzol and the generalists use different adaptation methods and model-specific inference runtimes.

3.2 The production model’s training history

Production is one final weight snapshot whose ancestry includes five selected training stages. Each stage inherits earlier weights and changes the emphasis of subsequent training. The selected endpoints are shown below; they are not a count of all attempted experiments, all compute spent, or the base model’s pretraining.

Later recipes use replay examples and frozen-reference penalties to encourage retention while adapting. This history gives a concrete account of how the deployed specialist was produced. It does not establish that this order is optimal or that every stage was necessary. The comparison contains no matched single-stage, equal-exposure ablation that would isolate the effect of staging.

3.3 One execution setup for all saved answers

Findbench 3 revision 3 supplies 986 search cases: 320 document, 160 filename, 266 media and 240 synthetic email cases. Sixteen model-refusal diagnostics are excluded. A model’s first answer is run against the corresponding fixture, and the outcome must match the expected paths or message IDs and matched count. Application sandbox and write-denial protections remain part of file execution.

The original large-model measurements used an earlier file binary and a Python mail executor. For this paper, we re-executed all four models’ saved search answers using the same frozen production file broker and native mail runner. This replay executed 4 × 986 = 3,944 answers. It changed zero pass/fail outcomes for any model relative to its saved comparison receipts. Production’s replay also matched its November 8 baseline case by case. The replay made no new model calls and did not regenerate answers.

Here B is the finite board, x the request, F its fixture, E the fixed execution-and-scoring procedure and Y the expected result. This is executed board accuracy, not a claim that the board is an unbiased sample of all search traffic. Token loss, a model judge’s preference and textual similarity are not substitutes for this metric.

Saved external answers retain their recorded processing: outer whitespace removal, with Gemma’s thought channel separated from its final answer. The replay does not repair plans or select a better answer. Fixing the executor controls an important measurement difference; it does not make the original inference prompts, runtimes or training budgets identical.

4. Results: close file accuracy, different strengths

Tofazzol passes two more file cases than Gemma and four more than Flash, differences of 0.27 and 0.54 percentage points. It passes 38 more than Flash-Lite, a 5.09-point difference. These are descriptive differences on this board. We did not preregister an equivalence or noninferiority margin, so “comparable” is not the outcome of a formal equivalence test.

The file aggregate contains a tradeoff. Gemma passes two more document cases and three more filename cases than Tofazzol. Tofazzol passes seven more media cases, giving its net two-case advantage. Against Flash, the corresponding differences are −1 document, −3 filename and +8 media cases. Tofazzol therefore does not lead every file category.

Email changes the broader ranking. Gemma and Flash each gain all 23 email cases that production misses and lose none on that board. Flash-Lite passes 233/240, versus production’s 217/240. Consequently, Gemma and Flash lead the combined search total at 973/986 and 971/986; production reaches 952/986, ahead of Flash-Lite’s 930/986. The file-search result supports task-specific competitiveness, not equal performance across the entire product.

4.1 Similar totals do not imply interchangeable behavior

Matching a score is less demanding than matching every behavior. The paired counts show that replacing one model with another can break working searches even when the total rises. They also show complementary successes, but a deployable router is not established by looking at which answer was correct after the fact. Any routing rule would need its own frozen evaluation without access to the expected answers.

5. How specialization can close the gap

The result becomes less surprising when we separate general model capacity from the requirements of a particular interface. The mechanisms below explain why the architecture and training recipe could work. Their presence is documented; their individual causal contributions are not measured by this comparison.

5.1 Pretraining supplies a foundation; fine-tuning concentrates practice

The small model already has learned language and code representations before this project begins. Fine-tuning can concentrate subsequent updates on the application’s recurring distinctions: content versus filename, date versus sender, alternatives versus conjunctions, and exact identity versus semantic topic. This is supervised practice on a specialized mapping.

A generalist given 30 demonstrations adapts through its input context. The specialist has received many gradient updates across task examples. Zero demonstrations at inference does not mean zero task experience. The comparison answers a useful deployment question about these complete recipes, while leaving open what the larger models would do after comparable task-specific fine-tuning.

5.2 A small output language reduces what must be generated

The model produces a compact plan with a limited set of supported fields and operators. It does not need to write a long explanation, implement a new search algorithm or memorize the documents being searched. Repeatedly mapping requests into this interface can make a subset of language behavior reliable with a smaller model.

This resembles semantic parsing, where language is translated into executable representations. It should be distinguished from grammar-constrained decoding. PICARD, for example, rejects inadmissible tokens during generation through incremental parsing. [9] The reported Tofazzol setup does not implement PICARD; it learns the output format and the executor checks the resulting plan. A constrained contract and a constrained decoder are different mechanisms.

5.3 The system delegates exact work to software

The executor contributes substantial capability. The model chooses what to search for, while software performs exact matching and filtering over the available records. This arrangement lets a compact model occupy the language-interpretation role inside a larger engineered system.

Belcak et al. argue that many repeated, specialized calls in agentic systems can be served by smaller models, with broader models retained where their capabilities are needed. [7] Tofazzol supplies a concrete case consistent with that position. Our experiment does not establish that all agent tasks admit the same substitution.

5.4 Execution feedback makes errors actionable

Exact result checks turn vague model quality into identifiable failures. A copied character, a field assignment or a unit conversion can be connected to a specific missed or extra result. Training data and preservation objectives can then target those behaviors. Near-miss fixtures are especially useful because a broad, insufficiently constrained plan should retrieve the wrong set.

Execution remains only as discriminating as its fixtures. A numerically inaccurate size threshold can pass if no fixture record lies near the disputed boundary. Work on semantic evaluation for text-to-SQL explains why multiple discriminating database instances can provide stronger evidence of meaning than agreement on one instance. [11] The same concern applies to search plans.

5.5 Staged adaptation may preserve useful behavior, but it is not a guarantee

Production’s ancestry alternates broad learning and narrower refinement, with replay and reference penalties in later recipes. Such mechanisms can discourage changes to earlier behavior while adding new behavior. The selected model shows that a successful endpoint exists; it does not prove that the sequence itself caused the success.

The companion study illustrates the limit: a later continuation learned 599/600 seen email examples and reached 199/200 on related synthetic development cases, while the main email total stayed at 217/240. More training data and better fit did not automatically improve the desired evaluation. The companion empirical paper documents those runs and their gains and losses.

6. Specialization, distillation and compression

These terms describe related but distinct ideas. Specialization is adaptation to a particular task distribution. Distillation transfers behavior from a teacher to a student through a training signal. Quantization reduces the precision used to store or compute parameters. A small specialized model can be produced with or without teacher distillation, and quantizing a 31B model does not change its parameter count to 1.5B.

Hinton, Vinyals and Dean describe distillation as a way to transfer useful behavior from expensive models into a more deployable model. [5] This provides conceptual background for moving some expense from deployment to training. The Tofazzol comparison is not a replication of that method, nor a demonstration that a 1.5B model has compressed all of Gemma’s knowledge.

Hsieh et al.’s Distilling Step-by-Step is a particularly relevant precedent: teacher rationales provide additional supervision for small task-specific models. They report a 770M T5 model outperforming a few-shot 540B PaLM comparator on one benchmark with 80% of the available training data. [6] That is evidence that this broader phenomenon predates our experiment. Tofazzol’s single-plan supervision and preservation recipes do not reproduce their rationale-based multi-task method.

Self-Instruct provides another connection: generating and filtering instruction examples before fine-tuning. [8] For a search specialist, executable checks can strengthen data admission. They establish properties of the admitted examples; they cannot establish that the example distribution covers all natural user requests. The later Gemma-authored email experiment makes that distinction visible.

7. Why the result matters for local deployment

The production weight file occupies 3,087,467,144 bytes, approximately 3.09 GB in decimal units, with 338 BF16 tensors. Google’s cited Gemma dense-model count is approximately 30.7B parameters, about 19.9 times the measured Tofazzol count. A simple two-byte-per-parameter estimate for that count is about 61.4 GB of weights. This arithmetic is not a measurement of runtime memory; caches, activations, implementation details and multimodal components can change the required memory.

That difference makes the local deployment question concrete. If a compact model supplies sufficient accuracy for a repeated task, the application can execute its model calls on the user’s machine. The cloud API call is absent from that model path, and the search request can remain local. Local operation still uses hardware, electricity and engineering effort.

We do not claim a twentyfold speedup, twentyfold energy reduction or a measured total-cost advantage. The model inference runs used different hardware, concurrency and serving software, and this replay measures correctness of saved plans. A fair latency or cost study would need a separate protocol. The demonstrated deployment advantage here is the small, measured weight footprint alongside competitive executed file-search accuracy.

8. Boundaries of the evidence

Development-board adaptation. The board was used repeatedly during development and contains related scenario families. The local model was selected through extensive experimentation, while the generalists received task-covering demonstrations. The scores are not independent confirmation on a randomly sampled usage distribution. Work on adaptive data analysis explains why repeated evaluation-guided choices weaken later inferential claims. [12]

Structural generalization. No exact demonstration/request duplication is useful to check, but familiar query structures can remain across different names and wording. Finegan-Dollak et al. show why text-to-SQL evaluation should distinguish those forms of generalization. [10] This paper does not claim that held-out wording or new entity strings alone establish a new capability.

Coverage gaps. The 240 email-plan cases do not cover retrospective Sent reply/forward retrieval. In a separate generated 100-query real-cache comparison, production retrieved 83 selected targets and returned 72 exact complete results; all ten subject-plus-calendar requests failed. Reference completeness was limited to 91 cases. Those findings, described in the companion paper, prevent a broad real-email reliability claim. The three generalists were not run on that private-mail expansion, so it provides no real-mail ranking against them.

Unequal recipes. We compare full fine-tuning with few-shot prompting. There is no control for architecture, pretraining, total task exposure or optimization budget. Generalist instruction and demonstration choices can affect the outcome. An equally fine-tuned larger model could perform differently; this study does not estimate its ceiling.

Accuracy and statistical uncertainty. The two- and four-case file differences are small. No formal equivalence result or significance claim is made, and related scenario cases should not be treated as fully independent samples. The paired counts support inspection of concrete differences without turning them into a population-wide ranking.

Reproducibility and privacy. This artifact includes aggregate outcomes, paired counts, generation settings, figures and identity hashes. Private fixtures, request sets, predictions, training corpora and weights are withheld. Outside readers can inspect the arithmetic and methods, but cannot independently rerun the private experiment from the public packet alone. A public synthetic reproduction suite would be valuable future work.

9. Experiments that would test the explanation

A stronger follow-up would separate the mechanisms that are combined here. These are proposed controls, not experiments reported as completed in this paper.

CheckList offers a useful way to organize the capability inventory before increasing the number of examples. [13] For this application, that inventory should include literal copying, field binding, dates, folders, negative constraints, source availability and language that resembles an action but actually requests retrieval. Increasing coverage is a different intervention from repeating more instances of a familiar pattern.

10. Conclusion

A 1.5B search specialist can reach file-search results numerically close to, and slightly above, strong general models under the tested recipes: 735/746 for production Tofazzol, 733/746 for Gemma 4 31B and 731/746 for Gemini 3.8 Flash. The same model trails them on email and loses some file cases they pass. Both observations are essential to the conclusion.

The interesting phenomenon is the concentration of useful behavior: a pretrained small model, trained for a recurring interface and coupled to exact software execution, can satisfy a narrow operational requirement without carrying the deployment footprint of a much larger model. This case establishes that such an endpoint is achievable in the measured system. Isolating why it works, and how broadly it transfers, remains the next empirical question.

Appendix A. Evidence identities and companion artifacts

The same-executor replay reproduced all recorded outcomes, including production’s 217/240 email result. An earlier runtime/context pairing of the same production weights scored 216/240; that historical value is not substituted into this replay. Model identity and inference context remain part of a result even when the weight file is unchanged.

Download the aggregate results or vector figures: Figure 1, Figure 2 and Figure 3. All figures are embedded in this HTML for offline reading. The companion synthetic-email training paper discusses the later training runs and real-cache evaluation in greater detail.

Data availability. No training rows, real mail, raw prompts, individual predictions, per-case identifiers or weight tensors appear in this packet. The paper’s source data export is restricted to aggregate statistics, protocol settings and hashes. Preparing this document did not publish or upload private data to a public destination.

{
  "case_sha256": "ba401b4ce611cf70ad252e3a4bbcd9dcde0700010b842767e77bd674a109da45",
  "demonstration_validation": {
    "exact_request_overlap": 0,
    "passed": 60,
    "total": 60
  },
  "executed_answers_replayed": 3944,
  "external_demonstrations_per_context": 30,
  "external_protocol_sha256": "589f554e1ca33a1ee484c0162d9e381dd0f3c0852ca1572e7d2729d89a339e51",
  "external_settings": {
    "gemini": {
      "candidateCount": 1,
      "maxOutputTokens": 65536,
      "temperature": 0,
      "thinkingConfig": {
        "includeThoughts": false,
        "thinkingLevel": "MEDIUM"
      }
    },
    "gemma": {
      "chat_template_kwargs": {
        "enable_thinking": true
      },
      "max_tokens": 12000,
      "skip_special_tokens": false,
      "temperature": 0
    }
  },
  "file_broker_sha256": "f2565cf789564f1792ac07546d23a1d4e8f4407db8ff3f062b297009026d4a04",
  "gemini_parameter_counts": "Not disclosed in the cited model cards; no size ratio assigned.",
  "gemma_model_card_parameters_approx": 30700000000,
  "gemma_parameter_ratio_approx": 19.887099523824844,
  "historical_external_failures": {
    "gemini-3.5-flash-lite": {
      "__failed__": 0,
      "blocked": 0,
      "empty": 0,
      "http": 0,
      "truncated": 0
    },
    "gemini-3.8-flash": {
      "__failed__": 0,
      "blocked": 0,
      "empty": 0,
      "http": 0,
      "truncated": 0
    },
    "gemma-4-31b-it": {
      "__failed__": 0,
      "blocked": 0,
      "empty": 0,
      "http": 0,
      "truncated": 0
    }
  },
  "independent_confirmation": false,
  "limitations": [
    "Adaptively reused developer-authored board with correlated scenario families.",
    "Fine-tuned specialist versus prompted generalists, not equal adaptation budgets.",
    "Same executor replay; original inference contexts and runtimes remain model-specific.",
    "No general-capability, statistical-equivalence, latency, energy or total-cost claim.",
    "External models were not evaluated on the private 100-query mailbox expansion.",
    "Only Gemini 3.5 Flash-Lite is measured here; other Flash-Lite versions are unmeasured."
  ],
  "lineage_selected_steps": {
    "broad": 500,
    "calendar_fields": 250,
    "file_refinement": 600,
    "joint_file_mail": 600,
    "literal_copy": 400
  },
  "local_demonstrations_per_request": 0,
  "models": [
    {
      "changed_from_historical": 0,
      "files": [
        735,
        746
      ],
      "key": "production",
      "model": "Tofazzol production Copy400",
      "scores": {
        "documents": [
          318,
          320
        ],
        "filenames": [
          155,
          160
        ],
        "media": [
          262,
          266
        ],
        "outlook": [
          217,
          240
        ]
      },
      "search": [
        952,
        986
      ],
      "unsafe_results": 0
    },
    {
      "changed_from_historical": 0,
      "files": [
        733,
        746
      ],
      "key": "g31",
      "model": "Gemma 4 31B",
      "scores": {
        "documents": [
          320,
          320
        ],
        "filenames": [
          158,
          160
        ],
        "media": [
          255,
          266
        ],
        "outlook": [
          240,
          240
        ]
      },
      "search": [
        973,
        986
      ],
      "unsafe_results": 0
    },
    {
      "changed_from_historical": 0,
      "files": [
        731,
        746
      ],
      "key": "f38",
      "model": "Gemini 3.8 Flash",
      "scores": {
        "documents": [
          319,
          320
        ],
        "filenames": [
          158,
          160
        ],
        "media": [
          254,
          266
        ],
        "outlook": [
          240,
          240
        ]
      },
      "search": [
        971,
        986
      ],
      "unsafe_results": 0
    },
    {
      "changed_from_historical": 0,
      "files": [
        697,
        746
      ],
      "key": "fl35",
      "model": "Gemini 3.5 Flash-Lite",
      "scores": {
        "documents": [
          310,
          320
        ],
        "filenames": [
          141,
          160
        ],
        "media": [
          246,
          266
        ],
        "outlook": [
          233,
          240
        ]
      },
      "search": [
        930,
        986
      ],
      "unsafe_results": 0
    }
  ],
  "native_mail_sha256": "815c744a6b0524be4b1da8cdea87ee84db05874f8da666a944cfaa778f8b7ee0",
  "new_inference": false,
  "paired": {
    "f38": {
      "all": {
        "both_fail": 3,
        "both_pass": 940,
        "production_gains": 12,
        "production_losses": 31
      },
      "email": {
        "both_fail": 0,
        "both_pass": 217,
        "production_gains": 0,
        "production_losses": 23
      },
      "files": {
        "both_fail": 3,
        "both_pass": 723,
        "production_gains": 12,
        "production_losses": 8
      }
    },
    "fl35": {
      "all": {
        "both_fail": 6,
        "both_pass": 902,
        "production_gains": 50,
        "production_losses": 28
      },
      "email": {
        "both_fail": 1,
        "both_pass": 211,
        "production_gains": 6,
        "production_losses": 22
      },
      "files": {
        "both_fail": 5,
        "both_pass": 691,
        "production_gains": 44,
        "production_losses": 6
      }
    },
    "g31": {
      "all": {
        "both_fail": 3,
        "both_pass": 942,
        "production_gains": 10,
        "production_losses": 31
      },
      "email": {
        "both_fail": 0,
        "both_pass": 217,
        "production_gains": 0,
        "production_losses": 23
      },
      "files": {
        "both_fail": 3,
        "both_pass": 725,
        "production_gains": 10,
        "production_losses": 8
      }
    }
  },
  "paper_date": "2026-11-08",
  "production_parameters": 1543714304,
  "production_storage_dtype": "BF16",
  "production_weight_bytes": 3087467144,
  "production_weights_sha256": "f69c89242cdf8de3662d89d09ed0d61441cfa908afb423886fd4574f550cf8d4",
  "protocol_sha256": "871d219e26c4416e35637ae1d5da24aaec0f569b538e632bd3e580e535244b51",
  "returned_model_versions": {
    "f38": {
      "gemini-3.8-flash": 1002
    },
    "fl35": {
      "gemini-3.5-flash-lite": 1002
    },
    "g31": {
      "not returned": 1002
    }
  },
  "schema": "tofazzol-specialist-paper-comparison-v1"
}

Appendix B. A short conceptual guide

References and reading guide

Begin with Hsieh et al. [6] for a direct experimental precedent to the small-model phenomenon. Then read Hinton et al. [5] for distillation and Belcak et al. [7] for the deployment argument. PICARD [9] clarifies how an output language can be enforced. Finish with the evaluation papers [10–13] to understand what a convincing comparison must control.

1. Binyuan Hui et al. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186.Connection: the pretrained model family supplies the language-and-code foundation. Specialized fine-tuning begins after substantial pretraining.
2. Google DeepMind. 2026. Gemma 4 model card. Accessed 8 November 2026.Connection: the official dense-model parameter count, model family and supported capabilities. The size ratio in this paper uses approximately 30.7B from this card.
3. Google DeepMind. 2026. Gemini 3.8 Flash model card. Accessed 8 November 2026.Connection: model identity and intended general capabilities. The search scores here come from our recorded experiment, not the vendor’s benchmark table.
4. Google DeepMind. 2026. Gemini 3.5 Flash-Lite model card. Accessed 8 November 2026.Connection: the specific Flash-Lite comparator. Its results should not be generalized to every Flash-Lite release.
5. Geoffrey Hinton, Oriol Vinyals and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv:1503.02531.Reading question: which behavior is transferred into the student, and how does the training target make that transfer possible?
6. Cheng-Yu Hsieh et al. 2023. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Findings of ACL.Reading question: why do teacher rationales provide more useful supervision than final labels alone in their experiments? Distinguish their method from Tofazzol’s plan-only targets.
7. Peter Belcak et al. 2025; revised 2026. Small Language Models are the Future of Agentic AI. arXiv:2506.02153v3.Reading question: which repeated agent calls need broad capabilities, and which have a stable, narrow interface? This is a position paper, not universal proof of small-model dominance.
8. Yizhong Wang et al. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL.Reading question: what can generating and filtering examples teach a model, and how should downstream transfer be checked?
9. Torsten Scholak, Nathan Schucher and Dzmitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. EMNLP.Reading question: what does enforcing valid syntax during generation solve, and which semantic mistakes can still remain?
10. Catherine Finegan-Dollak et al. 2018. Improving Text-to-SQL Evaluation Methodology. ACL.Reading question: is a test asking for a new structure, or familiar structure with new wording and entities?
11. Ruiqi Zhong, Tao Yu and Dan Klein. 2020. Semantic Evaluation for Text-to-SQL with Distilled Test Suites. EMNLP.Reading question: how can different queries accidentally return the same records on one fixture, and how can additional fixtures distinguish them?
12. Cynthia Dwork et al. 2015. Generalization in Adaptive Data Analysis and Holdout Reuse. arXiv:1506.02629.Reading question: what changes when earlier test results influence later training and selection decisions?
13. Marco Tulio Ribeiro et al. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. ACL.Reading question: which capabilities could still be missing from a model with a high aggregate score?