When Numerical Validation Fails, It Is Usually a Definition Problem

Numerical validation is commonly framed as a statistical or computational challenge. In practice, most validation failures originate earlier -- at the level of definition. This note argues that numerical disagreement becomes problematic only when it cannot be interpreted, that tolerances and benchmarks often mask rather than resolve ambiguity, and that validation systems fail primarily by neglecting to declare explicit obligations. Without definition, numerical systems produce numbers, not knowledge.

Numerical Disagreement Is Not the Same as Validation Failure

Numerical systems disagree routinely. Different solvers, discretizations, data sources, execution orders, and hardware architectures yield different results. This is not exceptional; it is expected.

Validation failure occurs only when disagreement cannot be interpreted.

A difference that can be classified, bounded, and attributed is not a failure -- it is information. A difference that cannot be explained is not necessarily large, random, or unstable; it is undefined.

Most validation pipelines conflate disagreement with failure and noise with uncertainty. As a result, numerical comparison becomes subjective: numbers differ, tolerances are adjusted, explanations are improvised, and nothing is conclusively learned.

Why Tolerances Do Not Resolve Ambiguity

Tolerances are intended to absorb numerical noise. In practice, they often absorb ignorance.

A tolerance answers only one question: how much deviation is acceptable. It does not explain why deviation occurred, whether it was expected, or whether it violates the hypothesis under test.

When a tolerance is violated, the system produces a binary outcome -- pass or fail -- without context. When a tolerance is satisfied, the system remains silent. In neither case is the cause of agreement or disagreement examined.

This leads to a familiar pattern: repeated tolerance tuning without convergence. The tolerance becomes a policy knob rather than a verification boundary.

Benchmarks Fail for the Same Reason

Benchmarks are often treated as ground truth. In reality, they are reference implementations with their own assumptions, defaults, and failure modes.

When a system disagrees with a benchmark, several explanations are possible: the system is wrong; the benchmark is wrong; the systems are solving different problems; the comparison itself is ill-defined.

Without an explicit declaration of what is held fixed and what is allowed to vary, these explanations are indistinguishable.

Benchmark comparison then devolves into trust-based reasoning: "this solver is well established," or "this method is industry standard." These are social signals, not technical ones.

Validation Requires Declared Obligations

Validation is not about checking outputs. It is about verifying obligations.

An obligation is a statement of what must remain invariant for numerical results to be comparable. Obligations may exist at multiple levels: model formulation, data semantics, numerical method, execution order, runtime environment, or hardware behavior.

If an obligation is not declared, its violation cannot be detected. If it cannot be detected, it cannot be classified. If it cannot be classified, it will be ignored or rationalized.

Most numerical systems fail validation not because they violate obligations, but because they never stated any.

Unclassified Failure Is Worse Than Explicit Failure

A system that fails loudly is debuggable. A system that fails silently is not.

When numerical disagreement is unclassified, teams resort to informal explanations: floating-point noise, randomness, implementation quirks, or "expected differences." Over time, this erodes confidence in the validation process itself.

Worse, unclassified disagreement accumulates. Results that cannot be fully trusted are still compared, averaged, or promoted. The system continues to produce numbers, but no longer produces knowledge.

Explicit failure is not a weakness. It is the precondition for learning.

What a Validation System Must Be Able to Say

At minimum, a validation system should be able to answer the following questions without rerunning the experiment:

• What hypothesis was being evaluated?
• What was held fixed, and what was allowed to vary?
• Which obligations were satisfied?
• Which obligations were violated?
• Why was this result considered valid, invalid, or inconclusive?

If these questions cannot be answered from persisted artifacts alone, validation has already failed -- regardless of numerical agreement.

The Cost of Undefined Validation

Undefined validation scales poorly. As systems grow more complex, modular, and distributed, implicit assumptions proliferate faster than any team's ability to track them.

At that point, validation becomes ceremonial. Reports are generated, checks are run, and outcomes are recorded -- but confidence does not increase.

This is not a tooling problem. It is a definition problem.

Conclusion

Numerical validation does not fail because numbers differ. It fails because disagreement cannot be interpreted.

Until validation systems treat definition as a first-class concern -- through explicit obligations, declared invariants, and classifiable failure modes -- numerical results will remain fragile, explainable only by authority or hindsight.

Precision without definition is not rigor. It is decoration.