Update features: done infer triton pipeline, return box and save output. Update readme: benchmarks, cmd ex...

Author: k9ele7en

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2021, Kim Nguyen
+Copyright (c) 2021, Kim Nguyen, NVIDIA CORPORATION
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

README.md

@@ -73,20 +73,34 @@ I0714 00:37:55.312507 1 http_server.cc:2906] Started Metrics Service at 0.0.0.0:
 ```
 Run infer by cmd: 
 ```
-$ python infer_triton.py -m='detec_trt' -x=1 --input='./images/image_1.jpg' -i='grpc' -u='localhost:8001'
-
-$ python infer_triton.py -m='detec_onnx' -x=1 --input='./images/image_1.jpg'
+$ python infer_triton.py -m='detec_trt' -x=1 --test_folder='./images' -i='grpc' -u='localhost:8001'
+Request 1, batch size 1s/sample.jpg
+elapsed time : 0.9521937370300293s
 ```
+#### Performance benchmarks: single image (sample.jpg), time in seconds
+- Triton server: (gRPC-HTTP): <br>
+
+    | Model format| gRPC (s)| HTTP (s) |
+    |-------------|---------|----------|
+    | TensoRT     | 0.946   | 0.952    |
+    | Torchscript | 1.244   | 1.098    |
+    | ONNX        | 1.052   | 1.060    |
+
+- Classic Pytorch: 1.319s
 
 #### Arguments
 * `-m`: name of model with format
 * `-x`: version of model
-* `--input`: input image/folder
+* `--test_folder`: input image/folder
 * `-i`: protocol (HTTP/gRPC)
 * `-u`: URL of corresponding protocol (HTTP-8000, gRPC-8001)
 * ... (Details in ./infer_triton.py)
 
-
+#### Notes:
+- Error below is caused by wrong dynamic input shapes, check if the input image shape is valid to dynamic shapes in config.
+```
+inference failed: [StatusCode.INTERNAL] request specifies invalid shape for input 'input' for detec_trt_0_gpu0. Error details: model expected the shape of dimension 2 to be between 256 and 1200 but received 1216
+```
 b. Classic Pytorch (.pth) inference:
 ```
 $ python test.py --trained_model=[weightfile] --test_folder=[folder path to test images]

README_Triton.md

@@ -58,8 +58,8 @@ There are multiple method to infer the detec pipeline, devided into 2 main types
 If you run the pipeline using single model format, run as below, no need environment installation as Triton server.
 ```
 $ cd inference
-Run infer_pipeline.py with target method (pth/onnx/tensorrt) and suitable arguments of that method. Use single format, for ex pth format:
-$ python infer_pipeline.py --method='pth' --input='./images/image_1.jpg'
+Run infer_triton.py with target method (pth/onnx/tensorrt) and suitable arguments of that method. Use single format, for ex pth format:
+$ python infer_triton.py -m='detec_pt' -x=1 --test_folder='./images'
 ```
 ## I. Setup environment and tools
 First, update your PIP:
@@ -225,12 +225,12 @@ Convert source model into target formats and copy into Triton's Model Repository
 ## III. Run the server and client to infer (included in .sh script):
 Run server in container and client in cmd
 ```
-$ sudo docker run --gpus all --rm -p8000:8000 -p8001:8001 -p8002:8002 -v <full_path_to/data/model_repository>:/models nvcr.io/nvidia/tritonserver:<xx.yy>-py3 tritonserver --model-repository=/models
+$ sudo docker run --gpus all --rm -p8000:8000 -p8001:8001 -p8002:8002 -v <full_path_to/model_repository>:/models nvcr.io/nvidia/tritonserver:<xx.yy>-py3 tritonserver --model-repository=/models
 
-For example, run on server with full path "/home/maverick911/repo/Triton-server-CRAFT-pytorch
-/data/model_repository":
-$ sudo docker run --gpus all --rm -p8000:8000 -p8001:8001 -p8002:8002 -v /home/maverick911/repo/Triton-server-CRAFT-pytorch
-/data/model_repository:/models nvcr.io/nvidia/tritonserver:21.05-py3 tritonserver --model-repository=/models
+For example, run on server with full path "/home/maverick911/repo/triton-server-CRAFT-pytorch
+/model_repository":
+$ sudo docker run --gpus all --rm -p8000:8000 -p8001:8001 -p8002:8002 -v /home/maverick911/repo/triton-server-CRAFT-pytorch
+/model_repository:/models nvcr.io/nvidia/tritonserver:21.05-py3 tritonserver --model-repository=/models
 
 +----------------------+---------+--------+
 | Model                | Version | Status |
@@ -244,18 +244,21 @@ I0611 04:10:23.080860 1 http_server.c9:2906] Started Metrics Service at 0.0.0.0:
 ```
 2. Infer by client in cmd (this repo), with method (triton), model name (<model_type>_\<format>), version (not required). For ex:
 ```
-$ cd Triton-server-CRAFT-pytorch/
-$ python infer_pipeline.py --method='triton' -m='detec_onnx' -x=1 --input='./images/image_1.jpg'
-
+$ cd triton-server-CRAFT-pytorch/
+$ python infer_triton.py -m='detec_trt' -x=1 --test_folder='./images'
+Request 1, batch size 1s/sample.jpg
+elapsed time : 0.9521937370300293s
 ```
 ```
-$ python infer_pipeline.py --method='triton' -m='detec_trt' -x=1 --input='./images/image_1.jpg'
+$ python infer_triton.py -m='detec_pt' -x=1 --test_folder='./images' -i='grpc' -u='localhost:8001'
+Request 1, batch size 1s/sample.jpg
+elapsed time : 1.244419813156128s
 ```
 -------
 Run server in container and client sdk in container:
 1. Start the server side:
 ```
-$ sudo docker run --gpus all --rm -p8000:8000 -p8001:8001 -p8002:8002 -v /home/maverick911/repo/Triton-server-CRAFT-pytorch/model_repository:/models nvcr.io/nvidia/tritonserver:21.05-py3 tritonserver --model-repository=/models
+$ sudo docker run --gpus all --rm -p8000:8000 -p8001:8001 -p8002:8002 -v /home/maverick911/repo/triton-server-CRAFT-pytorch/model_repository:/models nvcr.io/nvidia/tritonserver:21.05-py3 tritonserver --model-repository=/models
 
 +----------------------+---------+--------+
 | Model                | Version | Status |
@@ -271,4 +274,4 @@ I0611 04:10:23.080860 1 http_server.c9:2906] Started Metrics Service at 0.0.0.0:
 ```
 $ sudo docker run -it --rm --net=host -v <full_path/to/repo>:/workspace/client nvcr.io/nvidia/tritonserver:<xx.yy>-py3-sdk
 ```
-3. Use infer_pipeline.py as example above to run.
+3. Use infer_triton.py as example above to run.

figures/craft_example.gif

@@ -3,6 +3,7 @@
 import numpy as np
 import cv2
 import imgproc
+from icecream import ic
 
 # borrowed from https://github.com/lengstrom/fast-style-transfer/blob/master/src/utils.py
 def get_files(img_dir):
@@ -30,7 +31,7 @@ def list_files(in_path):
     # gt_files.sort()
     return img_files, mask_files, gt_files
 
-def saveResult(method='triton', img_file, img, boxes, dirname='./result/', verticals=None, texts=None):
+def saveResult(img_file, img, boxes, dirname='./result/', verticals=None, texts=None, method='triton'):
         """ save text detection result one by one
         Args:
             img_file (str): image file name
@@ -46,8 +47,9 @@ def saveResult(method='triton', img_file, img, boxes, dirname='./result/', verti
         filename, file_ext = os.path.splitext(os.path.basename(img_file))
 
         # result directory
-        res_file = dirname + "res_" + filename + '.txt' if method!='triton' else '_triton.txt'
-        res_img_file = dirname + "res_" + filename + '.jpg' if method!='triton' else '_triton.txt'
+        tmp_name = dirname + "res_" + filename
+        res_file = tmp_name + '.txt' if method!='triton' else tmp_name + '_triton.txt'
+        res_img_file = tmp_name + '.jpg' if method!='triton' else tmp_name + '_triton.jpg'
 
         if not os.path.isdir(dirname):
             os.mkdir(dirname)

file_utils.py

@@ -3,10 +3,10 @@
 """
 _____________________________________________________________________________
 
-This file main inference pipeline to Triton
+This file contains main inference pipeline to Triton
 _____________________________________________________________________________
 """
-
+from icecream import ic
 import sys
 import os
 import time
@@ -30,23 +30,21 @@
 
 from collections import OrderedDict
 
-import tritonclient as triton
+import triton_utils as triton
 
 def str2bool(v):
     return v.lower() in ("yes", "y", "true", "t", "1")
 
-parser = argparse.ArgumentParser(description='CRAFT Text Detection')
+parser = argparse.ArgumentParser(description='Triton inference pipeline for CRAFT Text Detection')
 parser.add_argument('--text_threshold', default=0.7, type=float, help='text confidence threshold')
 parser.add_argument('--low_text', default=0.4, type=float, help='text low-bound score')
 parser.add_argument('--link_threshold', default=0.4, type=float, help='link confidence threshold')
 parser.add_argument('--cuda', default=True, type=str2bool, help='Use cuda for inference')
-parser.add_argument('--canvas_size', default=1280, type=int, help='image size for inference')
+parser.add_argument('--canvas_size', default=1100, type=int, help='image size for inference')
 parser.add_argument('--mag_ratio', default=1.5, type=float, help='image magnification ratio')
 parser.add_argument('--poly', default=False, action='store_true', help='enable polygon type')
 parser.add_argument('--show_time', default=False, action='store_true', help='show processing time')
-parser.add_argument('--test_folder', default='/data/', type=str, help='folder path to input images')
-parser.add_argument('--refine', default=False, action='store_true', help='enable link refiner')
-parser.add_argument('--refiner_model', default='weights/craft_refiner_CTW1500.pth', type=str, help='pretrained refiner model')
+parser.add_argument('--test_folder', default='images/', type=str, help='folder path to input images')
 
 # triton server
 parser.add_argument('-v',
@@ -110,7 +108,7 @@ def str2bool(v):
 if not os.path.isdir(result_folder):
     os.mkdir(result_folder)
 
-def test_net(args, net, image, text_threshold, link_threshold, low_text, cuda, poly):
+def test_net(args, image, text_threshold, link_threshold, low_text, cuda, poly):
     t0 = time.time()
 
     # resize
@@ -120,13 +118,13 @@ def test_net(args, net, image, text_threshold, link_threshold, low_text, cuda, p
     # preprocessing
     x = imgproc.normalizeMeanVariance(img_resized)
     x = torch.from_numpy(x).permute(2, 0, 1)    # [h, w, c] to [c, h, w]
-    x = Variable(x.unsqueeze(0))                # [c, h, w] to [b, c, h, w]
-    if cuda:
-        x = x.cuda()
+    # x = Variable(x.unsqueeze(0))                # [c, h, w] to [b, c, h, w]
+    # if cuda:
+    #     x = x.cuda()
 
     # send request to triton server
-    y, feature = triton.triton_requester(args, x)
-    
+    y = triton.triton_requester(args, x)
+
     # make score and link map
     score_text = y[0,:,:,0].cpu().data.numpy()
     score_link = y[0,:,:,1].cpu().data.numpy()
@@ -158,19 +156,24 @@ def test_net(args, net, image, text_threshold, link_threshold, low_text, cuda, p
 
 
 if __name__ == '__main__':
-
+    t = time.time()
     # load data
     for k, image_path in enumerate(image_list):
         print("Test image {:d}/{:d}: {:s}".format(k+1, len(image_list), image_path), end='\r')
         image = imgproc.loadImage(image_path)
 
-        bboxes, polys, score_text = test_net(args, net, image, args.text_threshold, args.link_threshold, args.low_text, args.cuda, args.poly, refine_net)
+        bboxes, polys, score_text = test_net(args, image, args.text_threshold, args.link_threshold, args.low_text, args.cuda, args.poly)
 
         # save score text
-        filename, file_ext = os.path.splitext(os.path.basename(image_path))
-        mask_file = result_folder + "/res_" + filename + '_mask_triton.jpg'
-        cv2.imwrite(mask_file, score_text)
+        # filename, file_ext = os.path.splitext(os.path.basename(image_path))
+        # mask_file = result_folder + "/res_" + filename + '_mask_triton.jpg'
+        # cv2.imwrite(mask_file, score_text)
 
-        file_utils.saveResult(method='triton', image_path, image[:,:,::-1], polys, dirname=result_folder)
+        file_utils.saveResult(image_path, image[:,:,::-1], polys, dirname=result_folder, method='triton')
 
     print("elapsed time : {}s".format(time.time() - t))
+
+# Example cmd:
+# python infer_triton.py -m='detec_trt' -x=1 --test_folder='./images' -i='grpc' -u='localhost:8001'
+# python infer_triton.py -m='detec_onnx' -x=1 --test_folder='./images'
+# python infer_triton.py -m='detec_pt' -x=1 --test_folder='./images'

images/sample.jpg

@@ -14,4 +14,4 @@ onnxruntime-gpu
 nvidia-pyindex
 onnx_graphsurgeon
 
-#tritonclient[all]
+tritonclient[all]

images/sample_img.jpg

@@ -2,7 +2,12 @@
 Copyright (c) 2019-present NAVER Corp.
 MIT License
 """
+"""
+_____________________________________________________________________________
 
+This file contains classic Pytorch pth inference from forked repo
+_____________________________________________________________________________
+"""
 # -*- coding: utf-8 -*-
 import sys
 import os
@@ -166,7 +171,7 @@ def test_net(net, image, text_threshold, link_threshold, low_text, cuda, poly, r
         mask_file = result_folder + "/res_" + filename + '_mask.jpg'
         cv2.imwrite(mask_file, score_text)
 
-        file_utils.saveResult(method='pth', image_path, image[:,:,::-1], polys, dirname=result_folder)
+        file_utils.saveResult(image_path, image[:,:,::-1], polys, dirname=result_folder, method='pth')
 
     print("Done, elapsed time : {}s. Check at folder result/".format(time.time() - t))
 

infer_triton.py

@@ -1,6 +1,11 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
+"""
+_____________________________________________________________________________
 
+This file contains functions for Triton inference pipeline
+_____________________________________________________________________________
+"""
 # Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -26,7 +31,7 @@
 # OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
+from icecream import ic
 import sys
 import torch
 import numpy as np
@@ -313,6 +318,7 @@ def triton_requester(FLAGS, img_resized):
         else:
             this_id = response.get_response()["id"]
         print("Request {}, batch size {}".format(this_id, FLAGS.batch_size))
+
         tmp = postprocess(response, output_name, FLAGS.batch_size, max_batch_size > 0)
         y = torch.from_numpy(tmp)
     return y