Rounding and Saturation - 2023.2 English

In the final stage, the final values are converted to TT_DATA using TP_SHIFT, TP_RND and TP_SAT. TP_SHIFT performs the scaling as described elsewhere. TP_RND and TP_SAT determine the form of rounding and saturation applied on the downshifted value. The following tables describes the form of rounding and of saturation performed.

Table 45 Rounding and Saturation in FFT

Parameter Name	Type	Description	Range
TP_RND	Unsigned int	Round mode	0 to 3 are not supported. These modes perform floor or ceiling operations which lead to large errors on output 4 = rnd_sym_inf symmetrical to infinity 5 = rnd_sym_zero symmetrical to zero 6 = rnd_conv_even convergent to even 7 = rnd_conv_odd convergent to odd
TP_SAT	Unsigned int	Saturation mode	0 = unsaturated 1 = asymmetric saturation i.e +2^(N-1)-1 to -2^(N-1) e.g. +32767 to -32768 3 = symmetric saturation i.e +2^(N-1)-1 to -2^(N-1)+1 e.g. +32767 to -32767

Distortion caused by saturation will be possible for certain configurations of the FFT. For instance, with TT_DATA=cint32, it is possible for the sample values within the FFT to grow beyond the range of int32 values due to bit growth in the FFT algorithm. Saturation is applied at each stage (rank). In the final stage when TP_SHIFT is applied, saturation is also applied according to TP_SAT. Similarly, if the FFT is configured for TT_DATA=cint16, but insufficient scaling (TP_SHIFT) is applied, then sample values may exceed the range of int16 and so these too will be saturated in the final stage. For TT_DATA=cfloat, the FFT performs no scaling, nor saturation. Any saturation effects will be due to the atomic float operations returning positive infinity, negative infinity or NaN.