all unknown even in an abvious case

[arasharchor]

Hi,
@tpys
I tried a sample video with one sample image. However, the recognition is so bad.

This is the sample image which I have put in facedb folder:

I also tried with its aligned image as well:

and this is the ouput:

https://www.youtube.com/watch?v=2HYxFLRBMqM&feature=youtu.be

I have left configuration as the default.

I put messi.jpg in folder messi in facedb. only the first parts of the paths were modified in all_in_one code.

I1129 00:02:25.147276  4081 net.cpp:242] This network produces output fc5
I1129 00:02:25.147359  4081 net.cpp:255] Network initialization done.
facedb[i].second is /home/nvidia/projects/Face/face-everthing/trained_models/cl-face/data/facedb/messi/messi.jpg
facedb[i].first is messi
I1129 00:02:25.968385  4081 all_in_one.cpp:310] Detection, time: 172.515 ms
I1129 00:02:27.606887  4081 all_in_one.cpp:349] Alignment, time: 843.107 ms
I1129 00:02:27.629241  4081 all_in_one.cpp:368] Tracking, time: 22.2895 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:27.679678  4081 all_in_one.cpp:400] Recognition, time: 50.3692 ms
I1129 00:02:27.690768  4081 all_in_one.cpp:310] Detection, time: 0.000417 ms
I1129 00:02:28.265631  4081 all_in_one.cpp:349] Alignment, time: 574.795 ms
I1129 00:02:28.289674  4081 all_in_one.cpp:368] Tracking, time: 23.9814 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:28.338394  4081 all_in_one.cpp:400] Recognition, time: 48.6523 ms
I1129 00:02:28.348994  4081 all_in_one.cpp:310] Detection, time: 0.000417 ms
I1129 00:02:28.923471  4081 all_in_one.cpp:349] Alignment, time: 574.41 ms
I1129 00:02:28.940681  4081 all_in_one.cpp:368] Tracking, time: 17.1484 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:28.990877  4081 all_in_one.cpp:400] Recognition, time: 50.1297 ms
I1129 00:02:29.003504  4081 all_in_one.cpp:310] Detection, time: 0.000208 ms
I1129 00:02:29.578398  4081 all_in_one.cpp:349] Alignment, time: 574.829 ms
I1129 00:02:29.599717  4081 all_in_one.cpp:368] Tracking, time: 21.257 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:29.648763  4081 all_in_one.cpp:400] Recognition, time: 48.9808 ms
I1129 00:02:29.660403  4081 all_in_one.cpp:310] Detection, time: 0.000365 ms
I1129 00:02:30.233350  4081 all_in_one.cpp:349] Alignment, time: 572.88 ms
I1129 00:02:30.255623  4081 all_in_one.cpp:368] Tracking, time: 22.2109 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:30.305956  4081 all_in_one.cpp:400] Recognition, time: 50.2634 ms
I1129 00:02:30.328394  4081 all_in_one.cpp:310] Detection, time: 0.000105 ms
I1129 00:02:30.904494  4081 all_in_one.cpp:349] Alignment, time: 576.032 ms
I1129 00:02:30.926556  4081 all_in_one.cpp:368] Tracking, time: 22.0018 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:30.976146  4081 all_in_one.cpp:400] Recognition, time: 49.5225 ms
I1129 00:02:30.990725  4081 all_in_one.cpp:310] Detection, time: 0.000156 ms
I1129 00:02:31.562122  4081 all_in_one.cpp:349] Alignment, time: 571.311 ms
I1129 00:02:31.582857  4081 all_in_one.cpp:368] Tracking, time: 20.6622 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:31.632879  4081 all_in_one.cpp:400] Recognition, time: 49.9329 ms
I1129 00:02:31.643852  4081 all_in_one.cpp:310] Detection, time: 0.000156 ms
I1129 00:02:31.644037  4081 all_in_one.cpp:349] Alignment, time: 0.092759 ms
I1129 00:02:31.644068  4081 all_in_one.cpp:368] Tracking, time: 0.000261 ms
I1129 00:02:31.644084  4081 all_in_one.cpp:400] Recognition, time: 0.000261 ms
I1129 00:02:31.733472  4081 all_in_one.cpp:310] Detection, time: 78.316 ms
I1129 00:02:32.343708  4081 all_in_one.cpp:349] Alignment, time: 567.695 ms
I1129 00:02:32.360647  4081 all_in_one.cpp:368] Tracking, time: 16.8745 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:32.409165  4081 all_in_one.cpp:400] Recognition, time: 48.4544 ms
I1129 00:02:32.427084  4081 all_in_one.cpp:310] Detection, time: 0.000104 ms
I1129 00:02:32.997759  4081 all_in_one.cpp:349] Alignment, time: 570.605 ms
I1129 00:02:33.013270  4081 all_in_one.cpp:368] Tracking, time: 15.4452 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:33.063443  4081 all_in_one.cpp:400] Recognition, time: 50.1033 ms
I1129 00:02:33.074196  4081 all_in_one.cpp:310] Detection, time: 0.000416 ms
I1129 00:02:33.646067  4081 all_in_one.cpp:349] Alignment, time: 571.803 ms
I1129 00:02:33.680100  4081 all_in_one.cpp:368] Tracking, time: 33.9718 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:33.728981  4081 all_in_one.cpp:400] Recognition, time: 48.8129 ms
I1129 00:02:33.740669  4081 all_in_one.cpp:310] Detection, time: 0.000104 ms
I1129 00:02:34.308459  4081 all_in_one.cpp:349] Alignment, time: 567.72 ms
I1129 00:02:34.339998  4081 all_in_one.cpp:368] Tracking, time: 31.4776 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:34.390161  4081 all_in_one.cpp:400] Recognition, time: 50.0975 ms
I1129 00:02:34.404956  4081 all_in_one.cpp:310] Detection, time: 0.000104 ms
I1129 00:02:34.974102  4081 all_in_one.cpp:349] Alignment, time: 569.074 ms
I1129 00:02:34.997243  4081 all_in_one.cpp:368] Tracking, time: 23.0777 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:35.045864  4081 all_in_one.cpp:400] Recognition, time: 48.5535 ms
I1129 00:02:35.063544  4081 all_in_one.cpp:310] Detection, time: 0.000156 ms
I1129 00:02:35.063705  4081 all_in_one.cpp:349] Alignment, time: 0.099738 ms
I1129 00:02:35.120887  4081 all_in_one.cpp:368] Tracking, time: 57.1466 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:35.169749  4081 all_in_one.cpp:400] Recognition, time: 48.7956 ms
I1129 00:02:35.180408  4081 all_in_one.cpp:310] Detection, time: 0.000208 ms
I1129 00:02:35.180567  4081 all_in_one.cpp:349] Alignment, time: 0.098853 ms
I1129 00:02:35.237015  4081 all_in_one.cpp:368] Tracking, time: 56.4075 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:35.285601  4081 all_in_one.cpp:400] Recognition, time: 48.5192 ms
I1129 00:02:35.299492  4081 all_in_one.cpp:310] Detection, time: 0.000104 ms
I1129 00:02:35.299805  4081 all_in_one.cpp:349] Alignment, time: 0.178383 ms
I1129 00:02:35.299890  4081 all_in_one.cpp:368] Tracking, time: 0.00026 ms
I1129 00:02:35.300035  4081 all_in_one.cpp:400] Recognition, time: 0.000365 ms
I1129 00:02:35.459854  4081 all_in_one.cpp:310] Detection, time: 146.888 ms
I1129 00:02:36.103457  4081 all_in_one.cpp:349] Alignment, time: 574.383 ms
I1129 00:02:36.121099  4081 all_in_one.cpp:368] Tracking, time: 17.5671 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:36.169976  4081 all_in_one.cpp:400] Recognition, time: 48.7918 ms
I1129 00:02:36.180562  4081 all_in_one.cpp:310] Detection, time: 0.000313 ms
I1129 00:02:36.755723  4081 all_in_one.cpp:349] Alignment, time: 575.072 ms
I1129 00:02:36.775389  4081 all_in_one.cpp:368] Tracking, time: 19.6035 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:36.825590  4081 all_in_one.cpp:400] Recognition, time: 50.1164 ms
I1129 00:02:36.835532  4081 all_in_one.cpp:310] Detection, time: 0.000157 ms
I1129 00:02:37.414881  4081 all_in_one.cpp:349] Alignment, time: 579.265 ms
I1129 00:02:37.434569  4081 all_in_one.cpp:368] Tracking, time: 19.6115 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown
I1129 00:02:37.483476  4081 all_in_one.cpp:400] Recognition, time: 48.8149 ms
I1129 00:02:37.494029  4081 all_in_one.cpp:310] Detection, time: 0.000208 ms
I1129 00:02:37.494186  4081 all_in_one.cpp:349] Alignment, time: 0.096925 ms
I1129 00:02:37.559808  4081 all_in_one.cpp:368] Tracking, time: 65.5808 ms
tracking_info[i].class_id_ is -1
tracking_info[i].name_ is unknown

[tpys]

@smajida So sorry for delayed response and thank you for trying my code, you should put aligned_face in data/facedb/aligned_face.jpg, you can obtain through #7 as for the recognition result, i just use the trained sphereface model from author, but use myself modified mtcnn align method, so it will have some bad affect, eventhough you can adjust the score threshold in FaceClassifier, but i think it's not a ultimate solution.

[arasharchor]

@tpys thanks for your reply. no problem. If you notice my first message, I have already tried putting aligned face in the mentioned folder and I got the video results which I have put its link above. As you can see, not even a single time Messi is recogized which means zero accuracy in true positive. I am surprized because the main author of sphereface has put a video demo in his repository in which it is almost 100% accurate both in true positive and true negative. What could be the reason?

[anguoyang]

Hi@smajida, I have the same question, did you find the solution?
Hi@tpys, it seems that you use only the mtcnn to get the aligned face?

[tpys]

@smajida @anguoyang I don't exactly why? maybe the pre-trained model doesn't work well with my c++ mtcnn alignment, currently i am busy on a face alignment algorithm, when i got time, will retrain the sphereface model. if you figure it out, please let me known too, thanks guys.

[anguoyang]

Hi@tpys, I have checked the recognition part, it seems that your source code is different with the original matlab one:
function feature = extractDeepFeature(file, net)
img = single(imread(file));
img = (img - 127.5)/128;
img = permute(img, [2,1,3]);
img = img(:,:,[3,2,1]);
res = net.forward({img});
res_ = net.forward({flip(img, 1)});
feature = double([res{1}; res_{1}]);
end

I will try to convert the original matlab source into c++ and have a try on their original model, and then get back here.

[arasharchor]

Hi @anguoyang @tpys , perhaps author of Sphereface algorithm has put the video demo which was processed with Sphereface64 rather than 20 model. Is Sphereface64 pre-trained model or its script available? I can give it try that and report the output here. I tried to convert the matlab code to C++, but I have not made it. If I make it, I will definitely post it here so others can benefit from that. :)

[tpys]

@smajida @anguoyang definitely worth try sphereface64 model, but in the recognition part, i basic follow same principle, subtract mean, divide by std, flip input, concatenate output to form 1024dim features, then normalize the feature.
see below codes

    int SphereFace::extract_feature_or_identify(const std::vector<cv::Mat>& face_images,
                                                const std::vector<cv::Rect>& face_windows,
                                                const std::vector<std::vector<cv::Point2f>>& face_landmarks,
                                                std::vector<std::vector<float>>& features) {
        assert(face_windows.size() == face_landmarks.size());
        vector<float> input_blob;
        for(size_t i = 0; i < face_images.size(); ++i){
            if(face_windows.size() == 0){
                do_ensemble_ = false;
            }
            this->preprocess(face_images[i], face_landmarks[i], input_blob);
        }
        sphere20_->forward_pass(input_blob.data());
        int src_image_num = face_images.size();
        sphere20_->extract_feature(src_image_num, features);
    }

    void SphereFace::preprocess(const cv::Mat& src,
                               const std::vector<cv::Point2f>& landmarks,
                               std::vector<float>&blob_data){

        //resize
        Mat image = src.clone();
        image.convertTo(image, CV_32FC3, 1.0/128, -127.5/128);

        //NHWC to NCHW
        vector<cv::Mat> channels(3);
        cv::split(image, channels);
        for (auto &c : channels) {
            blob_data.insert(blob_data.end(), (float *)c.datastart, (float *)c.dataend);
        }


        if(do_flip_){
            Mat image_flip;
            flip(image, image_flip, 0);
            vector<cv::Mat> channels_flip(3);
            cv::split(image_flip, channels_flip);
            for (auto &c : channels_flip) {
                blob_data.insert(blob_data.end(), (float *)c.datastart, (float *)c.dataend);
            }
        }
    }

[macheski]

I have found the reason, look here #8.

[arasharchor]

@macheski
Great.
Could you please use a demo video like the one in the original author's repo and put the link to the demo video here?
E.g. this video:
https://youtu.be/xvzN_1m-y1Y

I have asked the original Author to provide the video script long ago, but he has not replied to this issue.
wy1iu/sphereface#55

I am wondering how one can the similar result.

[macheski]

If you want to test a video, only need to modify the code in all_in_one.cpp.
The codes modified 👍
#if 0 //original code
VideoCapture cap;
if (!cap.open(video_dir)){
cap.open(0);
}
#endif

// modified code
VideoCapture *pCap = new VideoCapture("/home/author/test.avi");//write your video path here;
//new VideoCapture(0); //to use local camera, use this code.
Of course, you need replace cap. with pCap-> in the other codes.
Run the code, and nothing special need to be done, you can get a demo like this:

[arasharchor]

@macheski
@tpys
@anguoyang
thanks for your reply. Actually I had already done that by the following code:

    string filename = video_dir + "/friends.mp4";
    cout << "Initialization of reading video!" << endl;
    VideoCapture cap(filename); 

const string facedb_dir = "/home/MyProjects/Face/face-everthing/trained_models/cl-face/data/facedb";
const string video_dir = "/home/MyProjects/Face/face-everthing/trained_models/cl-face/data";

Ross

Rachel

Monika

Joey

Chandler

Phoebe

here is the way I put the images in facedb folder:

facedb--
-- Joey/jo.jpg
-- Ross/ro.jpg
-- Rachel/ra.jpg
-- Monika/mo.jpg
-- Chanlder/ch.jpg
-- Phoebe/ph.jpg

here is the results:
https://youtu.be/eZEyDG3Hjm0
there are some serious issues:

first Joey is recognized but Ross is also recognized as Joey. afterwards, all of them including Joey is not recognized. It seems the first person in the folder of images is recognized and then this recognition stays valid for some more recognitions and then it is totally deleted.

There is many false face detection. As far as I have seen. MTCNN is very good in detection. So why detection quality is not good here?!

can you test the code on the following video:
https://www.youtube.com/watch?v=xvzN_1m-y1Y

I investigated the outputs which you can see below. It seems after the system recognizes Joey it does not recognition for a few frames.

I0119 13:23:09.173434 46126 net.cpp:255] Network initialization done.
name : Rachel
name : Monika
name : Joey
name : Ross
name : Chanlder
name : Phoebe
Initialization of reading video!
Initialization of saving video!
Reading video started!
Show video frames!
Show video frames done!
class id : 2
name : Joey
showing results!
showing results!
showing results!
showing results!
showing results!
showing results!
showing results!
showing results!
class id : -1
name : unknown
showing results!
class id : -1
name : unknown
showing results!
class id : -1
name : unknown
showing results!
class id : -1
name : unknown
showing results!

[arasharchor]

@anguoyang @macheski @tpys
I modified the code and here is the results.
https://youtu.be/-LpWwNw67zo
and another one:
https://youtu.be/sYfRen5OoZ8

BTW, Chanlder's color is mistakenly white which is similar to color of unknown class.

I have to try to rotate bounding boxes like the original author's video demo.
In general there are two main issues:
MTCNN implementation seems not to be correct. I will replace it with my own which works very well in c++ and fast.
However the second issue is more important: face alignment takes long which @tpys also has mentioned it in one still open issues. For which one can as @tpys has suggested to use dlib face alignment. there are also other options which I would like to investigate with my colleague @Qbicz
I will update you guys afterwards.

[facexin]

@macheski
Hi,I want to draw a bounding box around the face like you.What should I do ?Thanks a lot.