Why does my ONNX model give different results?

Which differences are expected

Two correct implementations of the same model rarely produce bit-identical floats. Fused operators, different matrix-multiplication kernels, parallel reductions and hardware with different instructions all change the order of additions. For float32 models, differences around 1e-5 to 1e-4 relative are common; they should not change a predicted class or a detection.

A difference that changes decisions, grows with depth, or appears only on some inputs is not rounding. Treat it as a bug until you can explain it.

Common causes, in the order to check them

Preprocessing

The most frequent cause. Resize method, normalization mean and std, RGB vs BGR, channel order (NCHW vs NHWC), dtype and value range must be identical. Save the preprocessed tensor and feed that same tensor to both models.

Postprocessing

Thresholds, softmax or sigmoid applied on one side only, argmax over a different axis, or non-maximum suppression with other parameters. Compare raw model outputs before comparing decisions.

Export mode

A model exported in training mode keeps dropout active and uses batch statistics in batch normalization. Call model.eval() before torch.onnx.export (with the TorchScript exporter you can also pass training=torch.onnx.TrainingMode.EVAL); editing the exported graph afterwards rarely restores the original numbers.

Tracing, control flow and dynamic shapes

Tracing records the path taken by the example input. Data-dependent branches and loops can be frozen to that path, and dimensions not declared dynamic at export are fixed to the example’s size — so a different batch size or resolution behaves differently or fails.

Operator semantics and opset

Some operators changed behaviour between opsets or are approximated by the exporter, for example resize/upsample coordinate modes and rounding. Check the exporter warnings and the opset you targeted.

Precision

float16, TF32 on recent GPUs and int8 quantization trade accuracy for speed; quantized models in particular can diverge noticeably. Compare float32 first.

Execution provider and fallback

CPU, CUDA, TensorRT and other providers use different kernels and may fuse or approximate operations. When a provider lacks a kernel, ONNX Runtime assigns that node to another provider, usually CPU, so one session can mix kernels.

Graph optimizations

ONNX Runtime rewrites and fuses the graph at session creation. Fusions change the order of operations and can shift results slightly.

Versions

Different versions of ONNX Runtime, the exporter or drivers can change kernels and defaults. Pin them and record them.

Non-deterministic kernels

Some GPU kernels use parallel reductions or atomic operations whose order varies between runs, so even the same session on the same input can differ in the last bits.

How to diagnose different results across execution targets

When the same ONNX model gives different output on another target — CPU vs GPU, ONNX Runtime vs TensorRT, a workstation vs a Jetson or another edge device, float32 vs float16 — work through the differences in this order. Each step removes one variable, so the cause is the last thing you changed.

• Establish a baseline: one reference target (usually ONNX Runtime on CPU in float32) whose outputs you trust, ideally checked against the source framework.
• Freeze preprocessing and inputs: save the preprocessed tensor once and feed that same file to every target.
• Confirm artifact identity: compare the SHA-256 of the .onnx file each target actually loaded; a re-export or an optimized copy is a different model.
• Confirm runtime versions: record the ONNX Runtime (or TensorRT) version, execution provider, driver and opset on each side.
• Compare precision: float32 vs float16, TF32 or int8 is a change of model behaviour, not just of speed.
• Compare graph optimizations: run both sides with the same optimization level, then vary it on one side only.
• Define a numerical tolerance before looking at the numbers, from the task and its decision threshold.
• Compare outputs with that tolerance, and compare the decision (class, box, score threshold) as well as the raw tensors.
• Isolate the first divergent intermediate tensor where possible.
• Decide: a difference inside the tolerance that does not change decisions is expected; one that changes decisions, produces inf/NaN or grows with depth is a failure to fix.

Symptoms and where to look first

These are directions to investigate, not diagnoses: the same symptom can have more than one cause.

Tiny differences (for float32, typically below about 1e-4 relative), same decisions

Floating-point order of operations: different kernels, fusions or providers. Usually expected.

Large accuracy drop on every input

Preprocessing, export in training mode, quantization or an unsupported conversion.

CPU correct, GPU or accelerator wrong

Execution provider kernels, reduced precision (float16, TF32, int8), overflow, or nodes silently placed on another provider.

PyTorch correct, ONNX wrong

The exported graph: tracing, dynamic shapes, operator semantics in the target opset, or preprocessing outside the model.

Only one runtime version fails

Operator implementation or compatibility in that version; pin versions and compare release notes.

Difference depends on input shape or batch size

Dynamic shapes fixed at export, broadcasting, padding or resize behaviour.

inf or NaN on one target only

float16 range (maximum 65504) or an unstable operation such as a sum of squares, exp or division.

Choosing a tolerance

Exact equality is the wrong requirement for floating-point inference across targets: two correct runs differ in the last bits, and a small numerical difference is not, by itself, a failure. Compare with a tolerance, and report a few numbers rather than one.

There is no universal tolerance. It depends on the model, the precision (float16 errors are orders of magnitude larger than float32 ones), the task, how close outputs sit to a decision threshold, and the runtime and hardware. Set it from what the output is used for, write it down with the result, and keep it the same when you compare targets.

import numpy as np


def compare(reference, candidate, rtol=1e-3, atol=1e-5):
    """How far `candidate` is from `reference`, and whether the decision changed."""
    ref = np.asarray(reference, dtype=np.float64)
    cand = np.asarray(candidate, dtype=np.float64)
    diff = np.abs(ref - cand)
    return {
        "finite": bool(np.isfinite(cand).all()),
        "max_abs": float(diff.max()),
        "mean_abs": float(diff.mean()),
        # Relative to the tensor's scale: element-wise relative error explodes near zero.
        "max_abs_over_scale": float(diff.max() / max(np.abs(ref).max(), 1e-12)),
        "allclose": bool(np.allclose(cand, ref, rtol=rtol, atol=atol)),
        "top1_agreement": float((ref.argmax(-1) == cand.argmax(-1)).mean()),
    }

Absolute error

|reference − candidate| per element. The max absolute difference finds the worst element; the mean absolute difference shows whether the error is everywhere or concentrated.

Relative error

Absolute error divided by |reference|. Meaningful for large values, misleading near zero: after a ReLU, a 1e-7 difference on a value of 1e-9 is a relative error of 100.

Error relative to scale

Max absolute difference divided by the largest |reference| in the tensor. A stable single number for comparing layers or targets.

rtol and atol

numpy.allclose passes when |candidate − reference| ≤ atol + rtol × |reference| for every element: rtol handles large values, atol handles values near zero.

Decision agreement

Whether the downstream decision is the same: top-1 class, detections above the threshold, a score crossing a cut-off. Often the number that matters.

CPU vs another execution provider: a tested example

We ran a small image classifier (three Conv + BatchNorm + ReLU blocks, global pooling and a fully connected layer, float32, opset 17) on a batch of eight 224×224 inputs, once with the ONNX Runtime CPU provider and once on an integrated laptop GPU through the DirectML provider, both in onnxruntime-directml 1.24.4 so that only the provider changed.

The logits were not bit-identical: the largest absolute difference was 2.4e-7 (3.6e-7 of the largest logit), well inside rtol 1e-3 / atol 1e-5, and every top-1 class matched. That is the normal float32 picture; these DirectML numbers belong to this test environment (one integrated GPU, one driver, one ONNX Runtime build) and should not be generalised to other GPUs; a CUDA, TensorRT or edge provider would differ by different amounts, so measure your own.

Check that the accelerator really ran the graph: when a provider lacks a kernel, ONNX Runtime places that node on the CPU without failing. The session option session.disable_cpu_ep_fallback makes session creation fail instead, which turns a silent mix of providers into an explicit error.

import numpy as np
import onnxruntime as ort

x = np.load("input.npy")  # the same preprocessed batch for every run
cpu = ["CPUExecutionProvider"]


def run(path, providers, level=ort.GraphOptimizationLevel.ORT_ENABLE_ALL, inputs=None):
    options = ort.SessionOptions()
    options.graph_optimization_level = level
    session = ort.InferenceSession(path, options, providers=providers)
    feed = {session.get_inputs()[0].name: x if inputs is None else inputs}
    names = [o.name for o in session.get_outputs()]
    return dict(zip(names, session.run(None, feed))), session.get_providers()


accelerators = ["TensorrtExecutionProvider", "CUDAExecutionProvider", "DmlExecutionProvider",
                "CoreMLExecutionProvider", "OpenVINOExecutionProvider"]
target = next((p for p in accelerators if p in ort.get_available_providers()), None)

reference, _ = run("model.onnx", cpu)
if target is None:
    print("this onnxruntime build has no accelerator provider")
else:
    # Refuse silent fallback: session creation fails if any node would need the CPU.
    strict = ort.SessionOptions()
    strict.add_session_config_entry("session.disable_cpu_ep_fallback", "1")
    ort.InferenceSession("model.onnx", strict, providers=[target])

    candidate, used = run("model.onnx", [target, "CPUExecutionProvider"])
    print(ort.__version__, used)
    print(compare(reference["logits"], candidate["logits"]))

Graph optimization levels

ONNX Runtime rewrites the graph when a session is created: ORT_ENABLE_BASIC folds constants and removes redundant nodes, ORT_ENABLE_EXTENDED adds fusions such as Conv + BatchNormalization, and ORT_ENABLE_ALL adds layout optimizations. ORT_DISABLE_ALL runs the graph as written.

On the same classifier, on the CPU provider, every level changed the logits compared with ORT_DISABLE_ALL — none was bit-identical — but only slightly: a maximum absolute difference of 1.2e-7 for basic and extended, and 5.7e-7 for all levels (under 1e-6 of the largest logit), with identical top-1 classes. Optimizations are a plausible source of last-digit differences, rarely of decision changes; if disabling them changes a decision, look for a fusion or kernel bug.

baseline, _ = run("model.onnx", cpu, ort.GraphOptimizationLevel.ORT_DISABLE_ALL)
for name in ["ORT_ENABLE_BASIC", "ORT_ENABLE_EXTENDED", "ORT_ENABLE_ALL"]:
    out, _ = run("model.onnx", cpu, getattr(ort.GraphOptimizationLevel, name))
    identical = np.array_equal(baseline["logits"], out["logits"])
    print(name, "bit-identical:", identical, compare(baseline["logits"], out["logits"]))

float32 vs float16: expected variation

Converting the classifier to float16 (onnxconverter-common, keeping float32 inputs and outputs) changed the logits by up to 7.3e-4 on the CPU provider — about 1.1e-3 of the largest logit, three orders of magnitude more than the float32 differences above — and no top-1 class changed. That is expected variation for float16: compare it with a looser tolerance (here rtol 1e-2, atol 1e-3) and with the decision.

A float16 test on the CPU provider is not a test of float16 arithmetic: for most operators the CPU provider has no float16 kernel and computes in float32, so it mostly measures the effect of float16 weights. Test float16 on the provider you will deploy to.

import onnx
from onnxconverter_common import float16

fp16_model = float16.convert_float_to_float16(onnx.load("model.onnx"), keep_io_types=True)
onnx.save(fp16_model, "model_fp16.onnx")

fp32, _ = run("model.onnx", cpu)
fp16, _ = run("model_fp16.onnx", cpu)
print(compare(fp32["logits"], fp16["logits"], rtol=1e-2, atol=1e-3))

float16 failure: same model, correct on CPU, inf on GPU

float16 tops out at 65504. A mean of squares — the core of a variance or normalization layer — overflows as soon as an activation exceeds about 256. With activations of 300, the float16 version of this two-node model returned 90000 on the CPU provider and inf on the GPU through DirectML; the float32 version returned 90000 on both (90000.01 on the GPU).

The CPU answer was right only because it computed in float32. This is the shape of many "CPU correct, GPU wrong" reports: the model is the same, and the target runs it in a precision that cannot hold an intermediate value. Fixes include keeping that subgraph in float32 (mixed precision), rescaling inputs, or using a numerically stable formulation.

from onnx import TensorProto, helper

# A mean of squares: the core of a variance or normalization layer.
graph = helper.make_graph(
    [helper.make_node("Mul", ["x", "x"], ["squared"]),
     helper.make_node("ReduceMean", ["squared"], ["mean_square"], keepdims=0)],
    "mean_square",
    [helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 1000])],
    [helper.make_tensor_value_info("mean_square", TensorProto.FLOAT, [])],
)
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)])
model.ir_version = 8
onnx.save(model, "mean_square.onnx")
onnx.save(float16.convert_float_to_float16(model, keep_io_types=True), "mean_square_fp16.onnx")

activations = np.full((1, 1000), 300.0, dtype=np.float32)  # an ordinary activation value
targets = [cpu] + ([[target, "CPUExecutionProvider"]] if target else [])
for providers in targets:
    for path in ["mean_square.onnx", "mean_square_fp16.onnx"]:
        out, _ = run(path, providers, inputs=activations)
        print(providers[0], path, out["mean_square"])  # 300 squared = 90000 > 65504, the float16 max

Find the first layer where two runs diverge

When outputs differ beyond tolerance, compare intermediate tensors in graph order and stop at the first one that exceeds the tolerance. ONNX Runtime returns any tensor listed as a graph output, so the method is: list every intermediate tensor as an output on both sides, run both with graph optimizations off so that fused tensors still exist, and compare tensor by tensor with the same tolerance.

On the mean-of-squares model, the method stops at "squared", where float16 already holds inf. On the classifier, float32 on the CPU against float16 on the GPU first exceeded 1e-3 of scale at the very first convolution — the expected signature of a precision change, not of one broken operator. With a strict 1e-6 tolerance, float32 on the CPU against float32 on the GPU first crossed it at the third convolution, still far inside float32 noise: the tolerance decides what counts as divergent.

def expose_all(src, dst):
    """A copy of the model with every intermediate tensor also listed as an output."""
    model = onnx.shape_inference.infer_shapes(onnx.load(src))
    outputs = {o.name for o in model.graph.output}
    model.graph.output.extend(v for v in model.graph.value_info if v.name not in outputs)
    onnx.save(model, dst)
    return [name for node in model.graph.node for name in node.output]  # graph order


def first_divergence(path_a, path_b, providers_a, providers_b, inputs=None, tolerance=1e-3):
    order = expose_all(path_a, "a_all.onnx")
    expose_all(path_b, "b_all.onnx")
    off = ort.GraphOptimizationLevel.ORT_DISABLE_ALL  # keep every tensor, no fusions
    a, _ = run("a_all.onnx", providers_a, off, inputs)
    b, _ = run("b_all.onnx", providers_b, off, inputs)
    for name in order:
        if name in a and name in b:
            result = compare(a[name], b[name])
            if not result["finite"] or result["max_abs_over_scale"] > tolerance:
                return name, result
    return None, None


accel = [target, "CPUExecutionProvider"] if target else cpu
print(first_divergence("mean_square.onnx", "mean_square_fp16.onnx", cpu, accel, activations))
print(first_divergence("model.onnx", "model_fp16.onnx", cpu, accel))
print(first_divergence("model.onnx", "model.onnx", cpu, accel, tolerance=1e-6))

Limits of comparing intermediate tensors

• Observing changes the run, so the comparison is not fully passive: listing a tensor as an output can prevent fusions that consume it and can change the precision it is stored in, and on the CPU the float16 copy of "squared" read inf even though the CPU computed it in float32 and returned 90000. Confirm a finding with an end-to-end run.
• Optimized and fused graphs have fewer intermediate tensors; providers such as TensorRT build their own fused engines, so names may not match one to one. Compare at the boundaries that do exist, and use the provider’s own tools for engine-level layers (NVIDIA Polygraphy can mark all outputs and compare layer by layer).
• Some providers cannot return every intermediate tensor or change its precision when exposed.
• For large models, compare a few tensors at a time and narrow down (cut sub-models with onnx.utils.extract_model, below) instead of exposing everything at once.

Compare on the exact same input

Most investigations go wrong because the two sides never saw the same tensor. Save one preprocessed input, run both models on it, and look at the maximum absolute difference before deciding anything.

import numpy as np
import onnxruntime as ort
import torch

x = np.load("input.npy")                       # already preprocessed
model.eval()
with torch.no_grad():
    reference = model(torch.from_numpy(x)).numpy()

session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
candidate = session.run(None, {session.get_inputs()[0].name: x})[0]

print("max abs diff:", np.abs(reference - candidate).max())
np.testing.assert_allclose(reference, candidate, rtol=1e-3, atol=1e-5)

Remove one variable at a time

• Compare on CPU first. If CPU matches and GPU does not, the execution provider or precision is the cause.
• Disable graph optimizations to see whether a fusion is responsible.
• Force float32 end to end before testing float16 or int8.
• Find the first diverging layer: expose intermediate tensors as outputs, or cut a sub-model, and compare layer by layer.
• Pin and record the ONNX Runtime version, the execution provider, the exporter version and the opset for every comparison.

import onnx.utils
import onnxruntime as ort

options = ort.SessionOptions()
options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
session = ort.InferenceSession("model.onnx", options,
                               providers=["CPUExecutionProvider"])

# Keep only the part of the graph up to an intermediate tensor:
onnx.utils.extract_model("model.onnx", "head.onnx",
                         input_names=["input"], output_names=["layer3_out"])

Same model, different hardware

Running one model on several targets — a cloud CPU, a GPU, an edge accelerator — multiplies these sources of difference: each target has its own runtime build, execution provider, supported precision and kernels. Without a record of which runtime, version, provider and hardware produced each output, a difference found weeks later is hard to explain.

Comparing two executions is only reliable when each one is recorded with the facts that explain it, captured at run time rather than reconstructed later:

• the model artifact and its SHA-256, plus the opset;
• the runtime, its version and the execution provider;
• the execution environment, and the hardware when it is observed or reported;
• the identity of the input (a hash of the preprocessed tensor) and the precision used;
• the output, with its hash, and the tolerance it was compared with;
• when the run started and finished.

Where Auvryn fits

Auvryn validates an uploaded ONNX model and test-runs it once on ONNX Runtime (CPU) in an isolated sandbox, reporting the runtime version, output shapes and first values. Every execution records the model version, the environment, runtime and hardware facts when they are known, each with its source, and every result artifact with its SHA-256. Comparing the same model across environments is something Auvryn does internally today; it is not a self-service feature, and edge environments are not offered to external users.

What this does not cover

Bit-for-bit reproducibility across hardware is generally not available for floating-point models; deterministic runtimes for specific hardware exist but are a separate topic. Large language model serving adds its own sources of non-determinism, such as batching and sampling, which this guide does not address.

The tested examples were run on 2026-10-21 with Python 3.11.9, numpy 2.4.6, onnx 1.23.2, onnxconverter-common 1.16.0, onnxruntime 1.31.0 (CPU provider) and onnxruntime-directml 1.24.4 (CPU and DirectML providers, on an integrated laptop GPU). The numbers quoted are from that model and that hardware; yours will differ. The PyTorch comparison uses the standard torch.onnx workflow and was not run here.

This guide does not cover TensorRT engines, CUDA, Jetson or other edge devices directly: the method applies to them, but the measured differences above do not.

Primary sources

• PyTorch: exporting to ONNXtorch.onnx export, dynamic shapes and known limitations.
• ONNX Runtime execution providersHow providers are chosen and how nodes are assigned between them.
• ONNX Runtime graph optimizationsOptimization levels and how to turn them off.
• onnx.utils.extract_modelCutting a sub-model to compare layer by layer.
• ONNX Runtime: float16 and mixed precisionconvert_float_to_float16, keep_io_types and keeping operators in float32.
• ONNX Runtime DirectML execution providerThe GPU provider used in the tested example.
• ONNX Runtime 1.16 release notesIntroduces the session.disable_cpu_ep_fallback session option.
• ONNX Runtime issue #19437A contributor confirms that the CPU provider computes most float16 operators in float32.
• numpy.allcloseHow rtol and atol combine.
• NVIDIA PolygraphyLayer-wise comparison of ONNX Runtime and TensorRT outputs.

Next steps

• How to validate an ONNX model before deploymentThe checker, the model’s contract, a test load and a parity check.
• How to prove which model version produced a resultWhat to record with every run.
• AI execution provenanceRecording what ran, where and on what, with the source of each fact.
• Upload an ONNX modelValidate it and test it in an isolated sandbox after sign-up.

Frequently asked questions