Configure the Quantize Strategy - 3.5 English

Three main configurable parts of the quantize tool are the quantize tool pipeline, what part of the model to be quantized and how to quantize them. Define all these thing in quantize_strategy. Internally, each quantize_strategy is a JSON file containing below configurations:

pipeline_config

These configurations control the work pipeline of the quantize tool, including some optimizations during quantization, e.g., whether to fold Conv2D + BatchNorm layers, whether to perform Cross-Layer-Equalization algorithm and so on. It can be further divided into optimize_pipeline_config, quantize_pipeline_config, refine_pipeline_config and finalize_pipeline_config.

quantize_registry_config

These configurations control what layer types are quantizable, where to insert the quantize ops and what kind of quantize op to be inserted. It includes some layer specific configurations and user-defined global configurations.

Below is an example configuration for the Conv2D layers:

{
  "layer_type": "tensorflow.keras.layers.Conv2D",
  "quantizable_weights": ["kernel"],
  "weight_quantizers": [
  {
    "quantizer_type": "Pof2SQuantizer",
    "quantizer_params": {"bit_width": 8,"method":0, "round_mode": 1, "symmetry": true, "per_channel": true, "channel_axis": -1, "narrow_range": False}
  ],
  "quantizable_biases": ["bias"],
  "bias_quantizers": [
  {
    "quantizer_type": "Pof2SQuantizer",
    "quantizer_params": {"bit_width": 8,"method":0, "round_mode": 1, "symmetry": true, "per_channel": false, "channel_axis": -1, "narrow_range": False}
  ],
  "quantizable_activations": ["activation"],
  "activation_quantizers": [
  {
    "quantizer_type": "FSQuantizer",
    "quantizer_params": {"bit_width": 8, "method":2, "method_percentile":99.9999, "round_mode": 1, "symmetry": true, "per_channel": false, "channel_axis": -1}
  ]
}

As you can see, by using this quantize configuration, you quantize the weight, bias and activations of the Conv2D layer. The weight and bias are using `Pof2SQuantizer`(power-of-2 scale quantizer) and the activation are using `FSQuantizer`(float scale quantizer). You can apply different quantizers for different objects in one layer.

Users can use dump_quantize_strategy to see get the JSON file of current quantize strategy. To make things simple, four types of built-in quantize strategies for common user cases are provided, which users can extend or override for their need, including:

• pof2s: power-of-2 scale quantization, mainly used for DPU targets now. Default quantize strategy of the quantizer.
• pof2s_tqt: power-of-2 scale quantization with trained thresholds, mainly used for QAT in DPU now.
• fs: float scale quantization, mainly used for devices supporting floating-point calculation, such as CPU/GPUs.
• fsx: trained quantization threshold for power-of-2 scale quantization, mainly used for QAT for DPU now.

model = tf.keras.models.load_model(‘float_model.h5’)
from tensorflow_model_optimization.quantization.keras import vitis_quantize
quantizer = vitis_quantize.VitisQuantizer(model)
quantizer.quantize_model(calib_dataset,
                         input_layers=['conv2'],
                         bias_bit=32, 
                         activation_bit=32, 
                         weight_per_channel=True)

Example codes below shows how to do it.

quantizer = VitisQuantizer(model)
# Dump the current quantize strategy
quantizer.dump_quantize_strategy(dump_file='my_quantize_strategy.json', verbose=0)

# Make modifications of the dumped file 'my_quantize_strategy.json'
# Then, set the modified json to the quantizer and do quantization
quantizer.set_quantize_strategy(new_quantize_strategy='my_quantize_strategy.json')
quantizer.quantize_model(calib_dataset)