Skip to content

Commit 86bb056

Browse files
luo-cheng2021zhangYiIntelceciliapeng2011mangguo321ilyachur
authored
MulticlassNms/MatrixNms: transformations and CPU implementation (openvinotoolkit#6653)
* init version, need revise: opset7 * add convert testcase * multiclass_nms support spec * init version * matrixnms support spec * init support for matrix_nms * impl matirx_nms * implemented multiclass_nms reference. TODO: more test cases. * support dynamic shape in test * update to spec 0611 * update to spec 0611 * fixes. * fix: now sort by class_id and score work. * fix clang check error * more test cases verified. * fixes in ref impl. * attribute nms_eta works * test cross_batch and output_type i32. * enable multiclass-nms cpu plugin fallback ngraph * keep topk typo * enable matrix-nms cpu plugin fallback ngraph * support sort_result_across_batch * Add matrix_nms unit test * Add cross batch test cases * fix typo * move multiclass to opset8 * move matrixnms to opset8 * Reference implementations for MulticlassNms and MatrixNms ops * fix name conflict * remove unused var sort_result_across_batch default set to false * avoid float overflow * fix clang check error * info for mac fail * change testcase due to unstable sort * nms add 'normalized' attribute * multiclass cpu test support 'normalized' * nms add 'normalized' attribute * fixes: 1. normalized support. 2. sort by score before keep_top_k inside a batch. * fixes: 1. normalized support. 2. sort by score before keep_top_k inside a batch. * fix sort order in matrix_nms * fix review comments * add matrix_nms MKLDNN extension layer * parallel in matirx nms * separate filtered_box * separate class_nms result * parallel in class * parallel in batch * partial new nms * partial remove useless function * debug & fix * debug in indexing * fix test cases * remove logging * fix code-style * fix typo * add matrix_nms extension * nms python api * remove unused testcases * refactor transformation * transform dynamic shape to static shape * Update inference-engine/src/transformations/include/ngraph_ops/nms_static_shape_ie.hpp Co-authored-by: Ilya Churaev <[email protected]> * remove register_pass call * [MKLDNN]migrate matrix_nms to MKLDNNNode * bug fix in matrix_nms * padding on matrix_nms * remove logging * test case refine * merged transform_matrix_nms branch * refine matrixnms testcase * multiclass nms cpu plugin implement for static shape, rebased on Reference implementations PR * rebase to new multi-classs transform provided by lc * Name style algin with matrix-nms * static shape padding style to batch inside,new unit test method, real classnum shape * fix format * fix ci error * multi-class NMS modification based on PR reviewer opinion: code format, copyright, delete unused include and funciton way * explicit template instantiation due to mac ci fail * Yi3/fix review (#16) * fix coding style * use parallel_for2d * fix ci fail * unify 'copyright 2021' * mkldnn_multiclass_nms node update based on PR review (#17) * [MKLDNN] apply suggestion for matrix_nms (#18) * fix bug * apply review comments * apply review comments * apply review comments * apply review comments * skip only Nms test, not MatrixNms MulticlassNms test Co-authored-by: Zhang Yi3 <[email protected]> Co-authored-by: jialipen <[email protected]> Co-authored-by: mangguo <[email protected]> Co-authored-by: Ilya Churaev <[email protected]> Co-authored-by: liubo-intel <[email protected]>
1 parent c0c2f2d commit 86bb056

31 files changed

+2325
-6
lines changed

‎inference-engine/src/inference_engine/cnn_network_ngraph_impl.cpp

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,9 @@
3535

3636
#include <transformations/low_precision/disable_convert_constant_folding_on_const_path.hpp>
3737

38+
#include <transformations/op_conversions/convert_multiclass_nms_to_multiclass_nms_ie.hpp>
39+
#include <transformations/op_conversions/convert_matrix_nms_to_matrix_nms_ie.hpp>
40+
3841
#include "ie_ngraph_utils.hpp"
3942
#include "exec_graph_info.hpp"
4043
#include "ie_itt.hpp"
@@ -389,6 +392,8 @@ CNNNetworkNGraphImpl::reshape(const std::map<std::string, ngraph::PartialShape>&
389392
::ngraph::pass::Manager manager;
390393
// resolves dynamism by replacing dynamic operation with static version
391394
manager.register_pass<::ngraph::pass::ConvertNMS5ToLegacyMatcher>(false);
395+
manager.register_pass<::ngraph::pass::ConvertMulticlassNmsToMulticlassNmsIE>();
396+
manager.register_pass<::ngraph::pass::ConvertMatrixNmsToMatrixNmsIE>();
392397
manager.register_pass<::ngraph::pass::DisableConvertConstantFoldingOnConstPath>();
393398
manager.register_pass<::ngraph::pass::ConstantFolding>();
394399
// OneHotToLegacy changes output precision

‎inference-engine/src/mkldnn_plugin/cpu_types.h

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,9 @@ enum Type {
8686
ExperimentalDetectronPriorGridGenerator,
8787
ExperimentalDetectronGenerateProposalsSingleImage,
8888
ExtractImagePatches,
89-
NonMaxSuppression
89+
NonMaxSuppression,
90+
MatrixNms,
91+
MulticlassNms
9092
};
9193

9294
enum Algorithm {

‎inference-engine/src/mkldnn_plugin/mkldnn_node.cpp

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,9 @@ static const InferenceEngine::details::caseless_unordered_map<std::string, Type>
225225
{ "ExperimentalDetectronPriorGridGenerator", ExperimentalDetectronPriorGridGenerator},
226226
{ "ExperimentalDetectronGenerateProposalsSingleImage", ExperimentalDetectronGenerateProposalsSingleImage},
227227
{ "ExtractImagePatches", ExtractImagePatches},
228-
{ "NonMaxSuppressionIEInternal", NonMaxSuppression}
228+
{ "NonMaxSuppressionIEInternal", NonMaxSuppression},
229+
{ "MatrixNms", MatrixNms},
230+
{ "MulticlassNms", MulticlassNms}
229231
};
230232

231233
Type TypeFromName(const std::string type) {

‎inference-engine/src/mkldnn_plugin/mkldnn_node.h

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,10 @@ static std::string NameFromType(Type type) {
194194
return "ExtractImagePatches";
195195
case NonMaxSuppression:
196196
return "NonMaxSuppression";
197+
case MatrixNms:
198+
return "MatrixNms";
199+
case MulticlassNms:
200+
return "MulticlassNms";
197201
default:
198202
return "Unknown";
199203
}

‎inference-engine/src/mkldnn_plugin/mkldnn_plugin.cpp

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,8 @@
5757
#include <transformations/op_conversions/simplify_ctc_greedy_decoder_seq_len.hpp>
5858
#include <transformations/op_conversions/convert_previous_nms_to_nms_5.hpp>
5959
#include <transformations/op_conversions/convert_nms_to_nms_ie_internal.hpp>
60+
#include <transformations/op_conversions/convert_multiclass_nms_to_multiclass_nms_ie.hpp>
61+
#include <transformations/op_conversions/convert_matrix_nms_to_matrix_nms_ie.hpp>
6062
#include <transformations/op_conversions/convert_deformable_conv_v8_to_v1.hpp>
6163
#include <transformations/smart_reshape/matmul_sr.hpp>
6264
#include <transformations/convert_precision.hpp>
@@ -168,6 +170,8 @@ static void Transformation(CNNNetwork& clonedNetwork, const Config& conf) {
168170
manager.register_pass<ngraph::pass::ConvertNMS3ToNMS5>();
169171
manager.register_pass<ngraph::pass::ConvertNMS4ToNMS5>();
170172
manager.register_pass<ngraph::pass::ConvertNMSToNMSIEInternal>();
173+
manager.register_pass<ngraph::pass::ConvertMulticlassNmsToMulticlassNmsIE>();
174+
manager.register_pass<ngraph::pass::ConvertMatrixNmsToMatrixNmsIE>();
171175
manager.register_pass<ngraph::pass::TransposeMatMul>();
172176
manager.register_pass<ngraph::pass::ConstantFolding>();
173177

0 commit comments

Comments
 (0)