Model Accuracy Test - 2.0 English

Vitis AI Library User Guide (UG1354)

Document ID
UG1354
Release Date
2022-01-20
Version
2.0 English

To test the model accuracy on the board, you must take into account the following factors.

  • Image dataset
  • Model
  • Accuracy test program
  • Ground truth file of the image dataset
  • Accuracy extraction comparison script

ResNet50 Example

Take resnet50 as an example.

  1. Get the image dataset and the ground truth file of the dataset.

    You can get the image dataset information from https://github.com/Xilinx/Vitis-AI/tree/master/models/AI-Model-Zoo. resnet50 uses the imagenet dataset.

  2. Get the model.
    wget https://www.xilinx.com/bin/public/openDownload?filename=resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz -O resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz
  3. Copy the model to the board.
    scp resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz root@IP_OF_BOARD:~/

    It includes resnet50 and resnet50_acc models.

  4. Untar it on the board.
    tar -xzvf resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz -C /usr/share/vitis-ai-library/models
  5. Set the path and image name of the image dataset to a file, such as image.list.txt.

  6. Run the accuracy test program on the board.
    cd ~/Vitis-AI/demo/Vitis-AI-Library/samples/classification/
    ./test_accuracy_classification_mt resnet50 image.list.txt resnet50.image.list.result
    Note: The accuracy test loads the resnet50_acc model besides the resnet50 model. The only difference between the two models is the model prototxt file.

    After the accuracy test program running finished, the result file resnet50.image.list.result will be generated.

  7. Copy the resnet50.image.list.result to the host.
  8. Run the corresponding accuracy extraction comparison script to get the final accuracy.
    python evaluation.py image.list.gt resnet50.image.list.result


    The following is the code of evaluation.py

    
    #!/usr/bin/env python
    
    
    import sys
    # argv[1] must groundtruth
    readfile=open(sys.argv[1], 'r')
    readfile1=open(sys.argv[2], 'r')
    
    
    dic_val={}
    
    
    m = 0
    for line in readfile:
        temp = line.strip('/').split()
        key = temp[0]
        value = int(temp[1])
        dic_val[key] = value
        m = m + 1
    
    
    n = 0
    for line1 in readfile1:
        temp = line1.strip('/').split()
        if temp[0] in dic_val and int(temp[1]) == dic_val[temp[0]]:
            # print int(temp[1]), dic_val[temp[0]]
            n = n + 1
    
    
    #print m
    #print n
    readfile1.close()
    readfile2=open(sys.argv[2], 'r')
    rate = float(n)/float(m)
    print("accuracy of top-5: ", rate)
    
    
    l = 0
    a = 0
    for line2 in readfile2:
        a = a + 1
        if (a%5 != 1) : continue
        temp = line2.strip('/').split()
        if temp[0] in dic_val and int(temp[1]) == dic_val[temp[0]]:
            l = l + 1
    rate1 = float(l)/float(m)
    
    Note: For the accuracy extraction comparison script, refer to https://github.com/Xilinx/Vitis-AI/tree/master/models/AI-Model-Zoo.