How AI models miss free answers across 38 slang items

Self-made slang questions that accidentally leaked their own answers

When we compiled our internal benchmark suite, the initial goal was very simple and pragmatic. We wanted to select which model handles Indonesian and Javanese most reliably before deciding on production infrastructure. Out of five question sets with a total of 351 items we developed, one that caught our attention most was the id-B set containing 38 slang items. This set includes 30 slang meaning items and 8 fake word items whose correct answer explicitly states that the word is unrecognized. This set was designed to test semantic understanding, but during the post-test audit process, we found a striking structural flaw. That flaw was not in the models, but in our own set.

This question set unintentionally provided free answers to anyone, or any system, that paid attention to the physical shape of the answer options without needing to understand the language meaning. Mechanical rules that do not read the question content at all, such as choosing the longest option or detecting a particular pattern, turned out to be capable of producing scores far exceeding random guessing. We call this phenomenon the blind baseline. The main question changed drastically halfway through. We were no longer asking how smartly models handle slang, but how much models utilized our carelessness in constructing distractors. The blind baseline measures what the set permits, not what the model does, and those two things are often mistakenly equated in automated evaluation literature.

Controlled A/B experiment to measure option length loopholes

To measure how far this format leakage affects final results, we designed a controlled A/B experiment on the id-B slang set dated 2026-08-02. Two arms were run with identical conditions in every way except for a single variable: the rule in the set compiler script that truncates options longer than 90 characters. Arm A acted as the leaked arm with 59 out of 152 options truncated, yielding a blind baseline of 35.7% and 228 calls and 228 scored rows from 2 models with 3 repetitions using option randomization seed 20260802. Arm B acted as the clean arm with 0 out of 152 options truncated, yielding a blind baseline of 25.0% with the exact same number of calls, scored rows, models, repetitions, and seed.

This experiment was run through our harness, a program that sends questions to models and records the results. The selection of two low-cost class models was done to ensure testing ran quickly and consistently. The option randomization seed was set at 20260802 in both arms to lock multiple-choice positions so they would not shift randomly beyond text length variable control. Arm A was deliberately reconstructed from the truncation rule because the original file was overwritten before the control was run and could not be recovered. That reconstruction yielded 59 truncated options with a percentage of 35.7%, weaker than the original file which contained 75 truncated options with a percentage of 39.7%. Nevertheless, this parameter difference was enough to create a sharp contrast between conditions where option length structure provided physical clues and conditions where that structure was flattened.

Harness bug disguised as data that nearly derailed the entire process

This experiment almost failed completely not due to model intelligence issues, but due to a hidden bug in our own measurement tool. The first run of testing produced asymmetric arms, where Arm A only recorded 226 scored rows while Arm B recorded 228 rows. In an experiment claiming identical conditions except for one variable, this two-row difference is the death knell for data validity. Initial suspicion pointed toward API instability, but deep investigation proved the cause was faulty error handling inside the harness.

OpenRouter passes upstream provider failures as HTTP 400 codes. Our overly rigid harness labeled all 400 responses as permanent errors, so those requests were never retried and the rows simply vanished from the run. That the label was wrong can be read directly from the data itself without additional assumptions. Items id-b1-003 and id-b1-026 were recorded as failing on the first repetition, then succeeding twice in a row on the next identically identical request. Syntactically flawed requests never behave that way. After 400 responses were moved to the temporary error class within the harness, Arm A was rerun and produced 228 out of 228 scored rows with zero errors and one retry. The token column was also corrected from zero to empty to avoid the false assumption that measurement had occurred.

AI models almost completely ignored the free answers we provided

Once the data was cleaned of harness bugs, the comparison results between the two arms provided findings that were both surprising and relieving. Pure random guessing on 4 multiple-choice options is 25.0%. The flawed Arm A provided 10.7 points or the equivalent of 4.1 items for free to anyone who chose based on option physical shape without reading the content. However, out of the 4.1 free items available before them, the models we tested took at most 1.00 item. This figure proves that models do not necessarily exploit raw format loopholes present in question sets.

The table above summarizes the final scores of both models in both testing arms.

The noise floor on this set containing 38 items was calculated to be at 2.63 points, which is the exact representation of a single item. Deepseek's score difference is at 1.75 points, which is below the noise floor so it should not be reported as a real effect of format leakage. Conversely, Gemini's score difference is precisely at 2.63 points or exactly one item, which is the smallest threshold still distinguishable from chance fluctuation. The flip side of this data must also be acknowledged, where not all forms of leakage are ignored by models. Section id-b3 containing unique phrases in its answer keys proved to attract model attention with a perfect score of 100.0% in several tests, showing that leakage inherent to specific text has a different appeal compared to positional leakage.

Real changes we adopted in subsequent question compilation

These findings permanently changed our team's workflow in producing benchmark suites. We no longer trust per-item verification done manually by human reviewers, because such verification proved to let cross-item systemic flaws pass that only become visible after all data is computed. Now, every new question set is required to pass a blind baseline audit before being tested on any commercial model. If a simple mechanical rule like choosing the shortest option or the first option succeeds in guessing more than a quarter of total questions, the set is immediately returned to the compilation desk to flatten answer option lengths and structures.

Our second operational change relates directly to error handling inside the testing harness. We strictly separate permanent errors originating from flawed request structures from temporary errors originating from service provider infrastructure failures. Ignoring this difference proved to nearly ruin our previous experiment's integrity. In addition, we now ensure that the token metric column on failed requests is given an empty value instead of zero, in order to maintain the honesty of raw data records. All these mitigation steps are now mandatory standards in our internal research repository before any dataset is declared fit to enter the publication stage.

Limitations of scope and testing methodology

Every research finding has its own limits, and we state them transparently without a defensive attitude. This testing only involved two models, and both came from low-cost model classes. High-level reasoning models were not tested at all in this experiment. The number of items in the id-b slang set is limited to only 38 items, so the noise floor of 2.63 points makes small effects undetectable by design, not purely by chance. Gemini's score difference precisely touching 2.63 points is at that threshold limit and should not be interpreted further than the minimum detection limit.

Arm A used in this analysis is a reconstruction from truncation rules, not the original file dated July 27 which was overwritten and could not be recovered. That reconstruction file produced 59 truncated options with a percentage of 35.7%, weaker than the original file containing 75 truncated options with a percentage of 39.7%. Standard deviation only moved on one model out of the two models tested, where deepseek recorded 0.00 on the clean arm and 1.24 on the leaked arm, while gemini remained stable at 0.00 on both arms. This observation is not enough to conclude that truncated options consistently increase model doubt. Section id-b3 still contains uncorrected phrase leakage so scores in that section cannot be fully trusted yet. Finally, the compilation of these items only involved one primary reviewer without formal inter-annotator agreement testing.