Terrible performance when not using test image

[dshahrokhian]

I have seen that all other people are facing the same issue I am: When you use an image different from the provided test.jpg, the frontalization doesn't work at all.

@dougsouza I was wondering if you think that the problem may lie somewhere in your repo, by having hard-coded specific parameters for such image, or in contrast it is a problem in the original matlab code/model.

Thanks,
Dani

[dougsouza]

@dshahrokhian, this code is the same as the original matlab code. I didn't do any modifications to the behavior of the program. If you try the original code you should see the same issues.

[dshahrokhian]

@dougsouza would you kindly try this image? I am starting to think that the problem lies within my dependencies.

My output:

Thanks,
Dani

[TalHassner]

@dshahrokhian We are aware of the frontalization failing on large photos. We're not sure why this happens (mostly because we haven't had the time to debug this, I'm afraid).

A potential reason for this may be landmark detection failure. The same thing can also affect smaller in case the face is hard to process. Another possible reason is numerical instabilities when computing the transformation (using OpenCV).

Please try resizing the image so that the face is roughly the same size as in the test image and try visualizing the detected landmarks to see if that helps.

I can say that we are working on eliminating the need for face landmark detection, and we're already seeing this doing better on challenging images where existing LM methods fail. You can follow updates on my webpage and see in particular our code for the face augmentation project which is based on Doug's code but generalizes it.

I hope this helps,

Tal

[dshahrokhian]

Thanks for your comment @TalHassner.

Just to clarify, the landmark detection works fine. You can see that in the visualization output of other users and mine.

A temporal fix to the pipeline would be, I am not sure if it's already present, including face detection techniques (such as 1, which is SOTA). You can just crop the face and then apply the frontalization. Nevertheless, I guess this doesn't solve the problem if the cropped area is still big, in which case you would need to reduce the resolution.

[TalHassner]

@dshahrokhian Yes, you are correct. Adding face detection and resizing to standard size would complete this pipeline. Since the code was developed and tested on existing benchmarks (LFW and Adience) this was not necessary, but for other applications, this modification can help and should be simple to add.

[pavibalaji]

Hello,
I agree with this too. Is there any way we can bring the image back to the original state after performing some operations on the frontalized face? Because the width of the frontalized face is higher. I kind of want to stitch it back to the original image.
Any help is appreciated!

[TalHassner]

@pavibalaji This is not as simple as it sounds.

The process is designed to map image pixels to the output (frontalized) view. In the other direction (frontalized to input) you'll get non-integer pixel coordinates, which would require you to extrapulate colors when assigning pixels with values. This would likely cause smoothing artifacts as well as leave holes in the output (these typically look like little black hairline cracks in the output view).

It would be too hard for me to explain more here but a more detailed description of the frontalization process and the code you're using is available in our FG paper from last year and on our more recent project page