The detail algorithm implemention is illustrated as below:
As it is shown in the aboved pictures, the whole orderTokenize have 4 functions and dataflow between these functions.
- scanStrategy: the function gets ac strategy from the input “used_orders” and compute the size according to ac strategy type.
- loadZigzag: the function lookups the created table to get zigzag according to its input orders stream.
- updateLehmer: the function computes Lehmer according to its input zigzag stream.
- updateToken: the function computes token value and context according to its lehmer stream.