The AI Passed the Test. Had It Seen the Answers?

By Jared Mastroianni

• October 18, 2026
• 50
• Business Storage, Guides, Self Storage

Evaluation evidence stays credible when the team improving the system is separated from the blind cases used to judge release readiness.

A clean evaluation score means little if facility cases leaked into training, prompt examples, retrieval, tuning, or repeated rehearsals.

An AI assistant gets every after-hours access case right in a release test. It distinguishes a failed credential from a locked schedule, keeps an unresolved identity match out of the “verified” column, and sends a safety-related exception to a person. The score looks ready for a launch meeting.

Then someone notices that most of the test cases came from last month’s prompt library. The same cases were used during demonstrations, copied into a tuning worksheet, and discussed with the team that wrote the instructions.

The system may still be capable. The test no longer establishes it.

This is evaluation contamination: information from the evaluation set reaches the model, its surrounding workflow, or the people optimizing it before the evaluation is complete. The leak can happen through training data, fine-tuning, retrieval, few-shot examples, prompt revisions, tool fixtures, repeated test runs, reviewer coaching, or synthetic paraphrases of the same underlying case.

Self-storage operators do not need a research laboratory to manage the risk. They need a controlled evaluation boundary: separate development work from release evidence, trace every facility case to its source, keep a blind set outside the optimization loop, and label scores according to what the evaluation can actually prove.

A score is evidence only when the test remained independent

Operators already understand the physical version of this problem. A manager who knows the exact day and time of an inspection can make one hallway look perfect. That does not prove the normal process is reliable. Likewise, an AI workflow can perform well on cases its designers have already studied without demonstrating that it will handle a new case under ordinary operating conditions.

The problem is broader than a model memorizing text. A release test measures the whole decision path:

• the model and resolved runtime;
• system and user instructions;
• retrieval sources and filters;
• tools, schemas, and default values;
• policy rules and thresholds;
• post-processing and reason-code mapping;
• human review instructions; and
• the test harness itself.

If any part of that path was tuned against the release cases, the score may describe rehearsal rather than generalization.

NIST's AI Risk Management Framework Playbook says test sets, metrics, and the tools used for testing, evaluation, verification, and validation should be documented.¹ It is voluntary guidance, not a self-storage standard. Its practical value is that an evaluation result should be reconstructable: which cases, which system build, which metric, which operator context, and which exposure history produced the score?

Contamination has more than one door

Teams often check only whether an exact test file appeared in model training. That is one path. A useful release review checks at least eight.

1. Pretraining or vendor training exposure. The provider may have trained on public or licensed material that contains the case, answer, or close derivative.
2. Fine-tuning exposure. Internal examples, corrections, or demonstrations may have entered a tuning set.
3. Prompt exposure. Test cases or answers may appear in system instructions, few-shot examples, prompt templates, or an operator playbook.
4. Retrieval exposure. The evaluation environment may retrieve a solved incident, answer key, annotated ticket, or summary that contains the expected result.
5. Tool-fixture exposure. Test databases, mocked APIs, default fields, or filenames may reveal the label.
6. Optimization exposure. People may revise prompts, thresholds, mappings, or review rules after seeing release-set failures.
7. Repeated-run exposure. A nominally blind set becomes development material after the team runs it often enough and remembers the cases.
8. Semantic-family exposure. A paraphrase, screenshot, synthetic rewrite, or extracted field may carry the same decision pattern even when the text is not identical.

NIST's Generative AI Profile specifically recommends documenting the assumptions and limitations of benchmarks, including actual or possible training/test cross-contamination.² The profile is cross-sector guidance. It does not dictate a particular metric or determine whether an internal facility evaluation is valid. It does make one boundary clear: benchmark limitations belong beside the score, not in a footnote discovered after release.

Assign every case a lifecycle state

Do not maintain one folder called “test cases.” Give each case exactly one current lifecycle state:

• Development: visible to builders and available for prompt, policy, tool, or workflow changes.
• Validation: used for choosing among candidate configurations; not used for final release evidence.
• Release holdout: sealed from builders until the candidate build, metric, and decision rule are frozen.
• Post-release monitor: collected after deployment to measure live drift or emerging conditions under approved safeguards.
• Retired: no longer valid because the case is obsolete, exposed, disputed, duplicated, or dependent on a superseded policy.

The state applies to the case family, not just one file. If an incident note, its human label, its screenshot, its generated summary, and three paraphrases all resolve to the same operating event, splitting them across sets does not create independence.

Use a stable family ID and record derivation. A case can move from release holdout to development after the release decision, but it cannot later return as “unseen” evidence for the same model family or decision path. Create new holdout material instead.

Seal the release set before the team needs a good score

A blind set works only if the operating process protects it.

Before evaluation, freeze:

• the candidate model, runtime, prompt, policy, retrieval configuration, tools, and human-review instructions;
• the evaluation objective and exact decision being tested;
• the case population and sampling method;
• the expected outputs and who is allowed to see them;
• the metric definitions, denominators, tolerances, and stop conditions;
• the permissions that prevent builders from reading holdout payloads or answer keys; and
• the hash or version ID of every released artifact.

Then run the evaluation in a controlled environment. Record who launched it, which build ran, which sources were reachable, whether the system had network or retrieval access, and whether any case or answer appeared in logs visible to the builders before the release decision.

NIST's Artificial Intelligence Technology Evaluation program uses blind data and a sequestered environment to reduce train/test contamination and support more objective assessment.³ That program is not a recipe for self-storage operations. The transferable design idea is separation: the people improving the system should not have unrestricted access to the evidence reserved for judging the final candidate.

For a small operator, “sequestered” may mean a controlled repository, two-person access, one evaluation owner who is not editing the prompt, and a fresh case pack revealed only after the build is frozen. The control needs to match the consequence. It does not need to imitate a federal test lab.

Build cases from decisions, not trivia

An evaluation can remain secret and still be useless. Cases must represent the decisions the system will face.

For an access-support assistant, the case families might include:

• correct tenant, correct facility, active access window;
• correct tenant, wrong facility;
• credential valid but schedule closed;
• payment state unknown;
• conflicting identity records;
• stale facility-hours source;
• gate event missing while camera coverage is unavailable;
• emergency language without enough evidence to classify the event; and
• a request that exceeds the assistant's authority.

Each case should define the observed facts, missing facts, governing source, allowed output, prohibited output, reason code, handoff condition, and expected abstention behavior. Include incomplete and conflicting evidence. A test that always supplies clean inputs rewards the wrong architecture.

Use facility diversity deliberately: different access systems, time zones, operating hours, connectivity states, unit types, and local workflows. Do not infer demographic traits or use protected classes as shortcuts. A case must be relevant to the decision and governed by approved data practices.

NIST's current TEVV-Athlon report is an initial public draft, not final guidance. It frames AI evaluation around organizational objectives, events, tools, and measurement concepts so assessments can be customized to real applications.⁴ That is useful here because “accuracy” is too vague. A self-storage evaluation should name the exact operating outcome: correct routing, bounded language, abstention under missing evidence, safe handback, or another defined behavior.

Track exposure before interpreting results

For every release case, calculate an exposure state before scoring:

• No known exposure: no match found in governed development, tuning, prompt, retrieval, fixture, or prior-run records.
• Possible exposure: lineage is incomplete or a semantic-family match cannot be ruled out.
• Confirmed indirect exposure: a derivative or same-event representation appeared in the optimization path.
• Confirmed direct exposure: the case, expected answer, or label was available to the model or builders.
• Unknown: the provider or internal lineage is insufficient to classify the case.

Do not collapse possible or unknown into clean. Report separate denominators. A 92% result on 100 cases is not the same evidence when 35 have possible exposure and 10 have confirmed exposure.

The 2024 research paper Evaluation Data Contamination in LLMs shows why simple matching is not enough: contamination is difficult to define, detection methods can produce false negatives, and effects vary by model and benchmark.⁵ It is a research result, not proof about any facility system or a universal detection method. The useful boundary is humility: contamination checks are evidence, not a guarantee of purity.

A fictional self-storage release review

Consider Maple Span Storage, a fictional facility created only for this example.

The team evaluates an access-support workflow on 30 fictional cases. It passes 28. The headline score is 93.3%.

The exposure review finds that six cases were copied from a prompt workshop, four were close rewrites of the same incidents, and two answer keys were reachable through the evaluation retrieval index. Those 12 cases are not independent release evidence. Five more have incomplete lineage because an older demonstration folder was never inventoried.

The team does not recalculate the score over the remaining 13 cases and call that conclusive. The sample is now too small and may no longer cover the intended facility conditions. It records the original result as contaminated — not valid for release, retires the exposed families, closes the retrieval leak, freezes a new candidate build, and commissions a fresh blind set from an evaluation owner outside the prompt-editing loop.

All facilities, people, cases, numbers, systems, scores, and actions in this example are fictional. They do not describe a deployment, customer, product result, or performance claim.

Keep release, publication, and live performance separate

A clean holdout result supports only the scoped release decision. It does not prove that the system will perform the same way at every facility, that human reviewers will follow the workflow, or that live inputs will match the evaluation distribution.

Preserve four states:

1. Evaluation-ready: case set, build, metrics, owners, and access boundaries are frozen.
2. Evaluated: the exact run completed and the exposure review is recorded.
3. Release-authorized: an accountable person accepted the evidence for a defined scope and period.
4. Live-verified: post-release monitoring shows the deployed path and operating context remain within the approved boundary.

Do not use a provider benchmark, internal demonstration, or one successful test run as all four states.

The operator tool: an AI Evaluation Independence Register

Use the accompanying AI Evaluation Independence Register to govern one release evaluation. The register captures the decision objective, frozen system build, case family, lifecycle state, source event, derivation, exposure checks, answer-key access, retrieval boundary, prior runs, contamination state, metric inclusion, release disposition, and evidence references.

Three operating rules keep it honest:

1. The case family is the unit of independence. Paraphrasing the same event does not create a new blind case.
2. The metric freezes before the score appears. Do not change denominators or tolerances to rescue a disappointing run.
3. Unknown exposure weakens the claim. It does not become clean because the release meeting is scheduled.

An AI system should earn release by handling unfamiliar facility conditions inside a defined authority boundary. If it has seen the answers, the team may have learned something useful during development. It has not yet proved readiness.