Shape's jaw fit abnormally

[Allen-lz]

This work is wonderful!!
And I also paid attention to your previous related work
https://github.com/tencent-ailab/hifi3dface,
I directly replaced hifi3dface/3DMM/files/HIFI3D++.mat with hifi3dface/3DMM/files/HIFI3D++.mat
The following error occurred when running:
f_sigma0 = (paras) / sigma # (500, 1)
ValueError: operands could not be broadcast together with shapes (526,1) (500,1)
Then I solved the above error using the following:
In optimization/rgbd/step3_prefit_shape.py
--> AI-NEXT-Shape
datas = h5py.File(os.path.join(modle_base, "shape_ev.mat"), "r")
ev_f = np.asarray(datas.get("ev_f")).reshape(-1, 1)
sigma_shape = np.sqrt(ev_f / np.sum(ev_f))

--> HIFI3D++
sigma_shape = np.sqrt(basis3dmm['EVs']/np.sum(basis3dmm['EVs'])) # (526, 1)

In hifi3dface/optimization/rgbd/RGBD_utils/AddHeadTool.py
--> AI-NEXT-Shape
ev_f = pca_info_h["ev_f"]
sigma_shape = np.sqrt(ev_f / np.sum(ev_f))

--> HIFI3D++
basis3dmm = scipy.io.loadmat("3DMM/files/HIFI3D++.mat")
sigma_shape = np.sqrt(basis3dmm['EVs'] / np.sum(basis3dmm['EVs']))

But the result is much worse:

Am I using HIFI3D++.mat incorrectly?

[haoxianzGit]

Hello，thanks for your interest.
Maybe the wrong 3D keypoints are used when calculating ‘trans_base_2_camera’ (function 'get_trans_base_to_camera') in step2*.py. You can test the following 3D keypoints:
mu_shape = np.reshape(np.float32(basis3dmm['mu_shape']), [-1,3])
mu_shape_kp = mu_shape[basis3dmm['keypoints'][0]]
Please check if the results of step2 and step3 are normal.

[Allen-lz]

Hello，thanks for your interest.
Maybe the wrong 3D keypoints are used when calculating ‘trans_base_2_camera’ (function 'get_trans_base_to_camera') in step2*.py. You can test the following 3D keypoints:
mu_shape = np.reshape(np.float32(basis3dmm['mu_shape']), [-1,3])
mu_shape_kp = mu_shape[basis3dmm['keypoints'][0]]
Please check if the results of step2 and step3 are normal.

thanks for the tip
I visualized np.reshape(np.float32(basis3dmm['mu_shape']), [-1, 3])
From the visual result, there is no problem.

I guess the problem might be mu_shape[basis3dmm['keypoints'][0]]
what's its role
Is it related to a1, a2, a3 in get_trans_base_to_camera?

[yikai28]

The same concern here, the result become worse after using HIFI3D++.... May we ask how to solve it><?