zihangjiang/dr-learning-for-3d-face

Implementation for paper "Disentangled Representation Learning for 3D Face Shape" CVPR 2019

dr-learning-for-3d-face's Issues

Couldn't find face_{}.dat in FacewareHouse Dataset?

Sorry.
It seems that something named face_{}.dat is needed but I couldn't find it in FaceWareHouse dataset I downloaded.
Thanks for your help :)

Got extremely large face when performing data augmentation

I got extremely large face when I implemented the data augmentation method described in the paper. Here is my code

import config
import numpy as np
from lib_dr import get_dr, get_mesh
from tqdm import tqdm
import open3d as o3d
import openmesh as om


def generate_coefficients(m):
    r = np.random.uniform(0.5, 1.2)
    thetas = np.random.uniform(0, np.pi / 2, m - 1)
    c = np.repeat(r, m)
    for i in range(len(thetas)):
        c[m - i - 1] *= np.sin(thetas[i])
        c[:m - i - 1] *= np.cos(thetas[i])
    return c


n_aug = 500
m = 5

data_path = 'data/CoMA/data/FW_140/train.npy'
result_path = 'data/CoMA/data/FW_140/train_dr_{}.npy'.format(n_aug)

train_np = np.load(data_path)

everyone_template = 'data/FWH/Tester_{}/Blendshape/shape_0.obj'
mean_face_path = 'DR-Learning-for-3D-Face/data/disentangle/Mean_Face.obj'

features = []

for i in tqdm(range(1, 141)):
    feat = get_dr.get_dr(mean_face_path, everyone_template.format(i))
    features.append(feat)

features = np.array(features)

template_mesh = om.read_trimesh(mean_face_path)
mesh = o3d.geometry.TriangleMesh()
mesh.vertices = o3d.utility.Vector3dVector(template_mesh.points())
mesh.triangles = o3d.utility.Vector3iVector(template_mesh.face_vertex_indices())
mesh.compute_vertex_normals()

aug_res = []
for i in tqdm(range(n_aug)):
    c = generate_coefficients(m)
    ids = np.random.choice(features.shape[0], m, replace=False)
    samples = features[ids]
    tmp = get_mesh.get_mesh(mean_face_path, np.tensordot(samples, c, axes=[0, 0]))
    tmp = tmp.reshape(-1, 3)

    o3d.visualization.draw_geometries([mesh])
    aug_res.append(tmp)

aug_res = np.array(aug_res)
aug_res = np.concatenate([train_np, aug_res], axis=0)
print(aug_res.shape)
np.save(result_path, aug_res)

Is it a normal phenomenon?

the srcipts for data augmentation

Hi, I've read your paper but I couldn't generate interpolated data by myself :( . Can you release srcipts for data augmentation method metioned in your paper? Thanks very much.

the difference of "train"and "special_train"

hi, your work is excellent and i am interested in it. when reading your codes, i can't figure out the difference of the two functions "net.special_train" and "net.train" due to my weak mathematical foundation. could you please tell me why you use the two functions to train your model. thank you very much.

Hi， I am very interested in you project. How to deal with the situation in which two meshs have different topological structure(different vertex number...)?

How to deal with the situation in which two meshs have different topological structure(different vertex number...)?

bad case generate by fusion net!

good job for me!
follow is result generate by fusion net

some bad case in result, plus result also have bad case, but id and exp is good, do you have any idea about it?
plus:

id:

exp:

and what is the meaning of
bias = 0.9 * (self.m_list + self.M_list) / (self.M_list - self.m_list)
i find 0.9-->1.0 can improve expression perform

def test_generate_one(self, id_net, exp_net):
        def I(z):
            z_mean, z_log_var, z = id_net.encoder(z)
            dr_id = id_net.decoder(z)
            return dr_id
        def E(z):
            z_mean, z_log_var, z = exp_net.encoder(z)
            dr_exp = exp_net.decoder(z)
            return dr_exp
        def F(y, x):
            return self.gcn_comp([y, x])
        def load_model():
            our_model.load_weights(('../model/our_model/our_model{}{}.h5').format(self.prefix, self.suffix))

        z = self.real

        our_model = Model(z, [I(z), E(z), F(I(z), E(z))])

        if self.load:
            load_model()
            print('!!!!!!!!!!!load fusion model success!!!!!!!!!!!')

        test_dr_feature = np.fromfile(os.path.join('dr_feature', 'Tester_106', 'dr_1.dat'))
        test_dr_feature = test_dr_feature[np.newaxis, :]
        
        test_dr_feature = normalize_fromfile(test_dr_feature, self.M_list, self.m_list)

        id_code = id_net.encoder.predict(test_dr_feature, batch_size=self.batch_size)[0]
        exp_code = exp_net.encoder.predict(test_dr_feature, batch_size=self.batch_size)[0]

        print('\n')
        print(id_code)
        print(exp_code)
        print('\n')

        bias = 0.9 * (self.m_list + self.M_list) / (self.M_list - self.m_list)

        norm_id = id_net.decoder.predict(id_code, batch_size=self.batch_size)
        norm_exp = exp_net.decoder.predict(exp_code, batch_size=self.batch_size)

        plus = norm_id+norm_exp+bias
        recon = self.gcn_comp.predict([norm_id, norm_exp], batch_size=self.batch_size) + bias

        id_feature = denormalize_fromfile(norm_id, self.M_list, self.m_list)
        exp_feature = denormalize_fromfile(norm_exp, self.M_list, self.m_list)
        plus_feature = denormalize_fromfile(plus, self.M_list, self.m_list)
        recon_feature = denormalize_fromfile(recon, self.M_list, self.m_list)

how to get the front_part_v.txt and mean face in a new dataset?

Can this method do expression transfer between raw image?

how to get "min_data.npy"and "max_data.npy" in "data/disentangle"?

The topology of a new dataset

Hello! I want to apply your method on a new dataset. But a question that bothers me is how to make all 3d faces in the new dataset have the same vertex structure, for example, they could be averaged by direct addition. I would appreciate if you could give me some advice.

the folder ‘data/disentangle/real_data’ ？

Hello，What does the folder ‘data/disentangle/real_data’ in line 395 of file src/model.py refer to？

module 'src.pyigl' has no attribute 'eigen'?

pyigl version ?

pretrained model?

Hi!

I think there was a pre-trained model(google drive link). Where did it go?

Thanks in advance :)

ps) Can you tell when an augmentation script will be released?

what is 'delta' in data_utils.py?

First of all, thank you for publishing a great research.

I wonder what Delta means.

In data_utils.py line 13, delta = np.array([1,0,0,1,0,1,0,0,0])?

Thanks in advance :)

error in V2M2

good job for me!
may be some error in V2M2 func, like follow

V2M2 generate bad mesh when run generate_mean_face, like follow

i guess new_v + V is need like V2M, V2M is work but also have some bad case, like follow

is it normal? can you give me some advice? thx~ a lot!

can you share facewarehosue dataset?

hello, good job!
can you share facewarehouse dataset? i can not get it from teacher zhou. my email is [email protected]. thx~ very much!

How to determine the identity and expression of the interpolated face?

Thanks for your code!
I want to know how to determine the ID and expression of the augmented data.
Due to the new faces are interpolated by several identity samples, these samples are belong to different subjects, so what is identity of new data?

When will you release srcipts for data augmentation method metioned in our paper? Thanks!

When will you release srcipts for data augmentation method metioned in our paper? Thanks!

FileNotFoundError: [Errno 2] No such file or directory: 'data/disentangle/test_data/Feature141.npy'

line 154 in '/src/data_utils.py' is wrong?

np.save('../data/disentangle/test_data',np.vstack(train_data)) should be changed to np.save('../data/disentangle/test_data',np.vstack(test_data)) ?