Inconsistent Results After ONNX Runtime Optimization

[FFchopon]

Describe the issue

I am encountering an issue where the output of the model after optimization using ONNX Runtime is inconsistent with the original model.

• Actual Behavior:

AssertionError: 
Not equal to tolerance rtol=0.001, atol=0.001

Mismatched elements: 12 / 1056 (1.14%)
Max absolute difference: 0.00350001
Max relative difference: 0.07692306
 x: array([0.42  , 0.42  , 0.42  , ..., 0.1155, 0.126 , 0.084 ], dtype=float32)
 y: array([0.42  , 0.42  , 0.42  , ..., 0.1155, 0.126 , 0.084 ], dtype=float32)

• Expected Behavior:
  The optimized model should produce identical results for all outputs when compared to the original model, within the specified tolerance.

To reproduce

1. Download the model
2. run the following script:

import onnx
import onnxruntime as ort
from onnxruntime.transformers import optimizer
import numpy as np

model_path = "20730.onnx"
optimized_model_path = f"./opt.onnx"
sess_options = ort.SessionOptions()
sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL

original_session = ort.InferenceSession(model_path, sess_options, providers=["CUDAExecutionProvider"])
input_data = {"v5_0": np.random.rand(55, 7, 1, 40).astype(np.float32)}
original_output_names = [output.name for output in original_session.get_outputs()]
original_result = original_session.run(original_output_names, input_data)
original_result2 = original_session.run(original_output_names, input_data)
for r1, r2 in zip(original_result, original_result2):
    np.testing.assert_allclose(r1, r2, rtol=1e-3, atol=1e-3)

optimized_model = optimizer.optimize_model(model_path, opt_level=99)
optimized_model.save_model_to_file(optimized_model_path)
optimized_session = ort.InferenceSession(optimized_model_path, providers=["CUDAExecutionProvider"])
optimized_output_names = [output.name for output in optimized_session.get_outputs()]
optimized_result = optimized_session.run(optimized_output_names, input_data)

for r1, r2 in zip(original_result, optimized_result):
    np.testing.assert_allclose(r1, r2, atol=1e-3, rtol=1e-3)

Urgency

No response

Platform

Linux

OS Version

Ubuntu 20.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

5c1b7cc

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

No response

[FFchopon]

After further analysis, I found that the output "output" of the model exhibits inconsistencies. Upon modifying the model to specifically analyze which node is causing the inconsistency and by outputting the values at each node, the inconsistencies disappears. This suggests that it might be a precision issue. Could anyone help me resolve this issue? @xadupre

[xadupre]

I suspect one QDQ pattern is introducing some discrepancies. CPU also gives some discrepancies. Did you try to remove Conv, with or AveragePool to see which one is causing discrepancies?