- File path of the model to quantize.
- File path of the quantized model.
- A calibration data reader. It enumerates calibration data and generates
inputs for the original model. If you want to use random data for a quick
test, you can set calibration_data_reader to None.
- QOperator: quantizes the model with quantized
- QDQ: quantize the model by inserting QuantizeLinear/DeQuantizeLinear on
the tensor. Supports only 8-bit quantization.
VitisQuantFormat.QDQ: quantizes the
model by inserting VAIQuantizeLinear/VAIDeQuantizeLinear on the
tensor. Supports more bit-width and configurations.
quantizes the model by inserting FixNeuron (composition of
QuantizeLinear and DeQuantizeLinear) on the tensor.
- For DPU devices, set calibrate_method to
'vai_q_onnx.PowerOfTwoMethod.MinMSE' to apply power-of-2 scale quantization.
The PowerOfTwoMethod has two supported methods currently: MinMSE and
NonOverflow. The default method is MinMSE.
- A list(string) object. Names of the start nodes to be
quantized. Nodes before these start nodes in the model are not optimized or
quantized. For example, this argument can skip some pre-processing nodes or
stop quantizing the first node. The default value is .
- A list(string) object. Names of the end nodes to be
quantized. Nodes after these nodes in the model are not optimized or
quantized. For example, this argument can skip some post-processing nodes or
stop quantizing the last node. The default value is .
- Specifies the types of operators to quantize, such as ['Conv'] to quantize
Conv only. It quantizes all supported operators by default.
- Quantize weights per channel. For DPU, this must be set to False as it
currently does not support per-channel.
- Quantize weights with 7 bits. For DPU, the reduce_range is not supported,
so this must be set to False.
- Quantization data type of weight. For DPU, this must be set to
QuantType.QInt8. For more details on data type selection, refer to https://onnxruntime.ai/docs/performance/quantization.html.
- List of nodes names to quantize. The nodes in this list are quantized only
when this list is None.
- List of nodes names to exclude. The nodes in this list are excluded from
quantization when it is None.
- Optimizes the model before quantization is going to be deprecated soon. It
is not recommended because optimization changes the computation graph,
making debugging quantization loss difficult.
- Option used for large size (>2GB) model. The default value is
- Key-value pair dictionary for various options in different cases.
Currently used pairs:
- Symmetrize calibration data for activations
(default is False). In PowerOfTwoMethod calibrate_method, it
should always set ActivationSymmetric as True.
- symmetrize calibration data for weights (The
default value is True). In PowerOfTwoMethod calibrate_method, it
should always set WeightSymmetric to True.
- By default, some latent operators, such as
maxpool and transpose, do not quantize if their input is not
quantized already. Setting to True to force such an operator
always quantizes input and generates quantized output. Also, the
true behavior could be disabled per node using the
- The default value is False for static mode. If
enabled, only MatMul with const B is quantized.
- The default value is False, which quantizes
floating-point weight and feeds it to the solely inserted
DeQuantizeLinear node. If True, it remains floating-point weight
and inserts QuantizeLinear/DeQuantizeLinear nodes to weight. In
PowerOfTwoMethod calibrate_method, QDQ should always appear as a
pair. Therefore, you need to add qdq pair to weight, and it
should always set AddQDQPairToWeight to True.