iacopomasi/face_specific_augm

Face Renderer to perform Domain (Face) Specific Data Augmentation

face_specific_augm's Issues

Do we have synthetic lighting rendering in this repo?

Do we have synthetic lighting rendering from "Dataset Augmentation for Pose and Lighting
Invariant Face Recognition" paper in this repo? If yes, can you point out how to use it? If no, do you any plan for implementing it?

MATLAB 3D Model Renderer

I try to obatain other pose model using your project "MATLAB 3D Model Renderer", but I find the OSG doesn't support .mat format model.
Now i have the 3d model(maybe from BFM), so how to use the model In your script, or in other words, how to convert the .mat file to the format that osg support?
Thanks

ValueError: too many values to unpack (expected 2)

Hi,

I am encountering the below error,
File "/programs/Face-Pose-Net-master/face_renderer/ThreeD_Model.py", line 64, in createMask
contours, hierarchy = cv2.findContours(eyemask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
ValueError: too many values to unpack (expected 2)
Can you help me fix this?

Thanks.

question on "extreme poses and viewing conditions" and resnetON = yes

Hi, hope all is well.
If I set resnetON = yes in the config file, is that all I need to do to set the config to default to feed the rendered images to your Resnet 101?
I noticed that when I set resnetON = yes, the rendered poses only range from 0 to about 40 and not full profile so where it says, ""This is a deep network trained to recognize faces appearing in extreme poses and viewing conditions" as description for Resnet 101, is this limited to about 0 to 40 and not full profile? If so, do you think transfer learning on rendered pose 75 or even other full profile (pose 90) images work?
Also, what is in-plane alignment?

landmark position in facemask

Hi,

in the 3D model, I can only see 3D landmark position (model_TD).

I can project the face region with facemask easily, but I can't see the landmark position in facemask.

thus I can't know the 2D landmark position in the output image.

Any idea to get 2D landmark position in the output image?

Question about 3D head model

Hello, I want to know what type of the standard head model is? Is it a depth, a point cloud or a mesh?

Project input landmark points?

Is it possible to project the input landmark points to the frontalized face? Thanks!

ValueError: too many values to unpack

Preloading all the models for efficiency
Loading pose model in model3D_aug_-00_00_01.mat
Traceback (most recent call last):
File "demo.py", line 117, in
demo()
File "demo.py", line 48, in demo
allModels = myutil.preload(this_path,pose_models_folder,pose_models,nSub)
File "/home/akshay/augmented/face_specific_augm-master/myutil.py", line 66, in preload
model3D = ThreeD_Model.FaceModel(this_path + pose_models_folder + pose, 'model3D', True)
File "/home/akshay/augmented/face_specific_augm-master/ThreeD_Model.py", line 15, in init
self.eyemask = self.getEyeMask(width=8,plot=False)
File "/home/akshay/augmented/face_specific_augm-master/ThreeD_Model.py", line 38, in getEyeMask
output1 = self.createMask(eyeLeft, width=width)
File "/home/akshay/augmented/face_specific_augm-master/ThreeD_Model.py", line 64, in createMask
contours, hierarchy = cv2.findContours(eyemask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
ValueError: too many values to unpack

going from profile to frontal

I used py2to3 to get a python 3 version and testing on some profile face images but it seems that when the image starts in profile, it does not rotate it towards frontal. I guess a GAN frontalization is what I need here. Or will Face-Pose-Net do this?

Error when 'background=no' in the config.ini file

Hi Dr. Masi,

Thanks for sharing your code. It works perfectly fine on my 512x512 face images (with 'resizeCNN' = 512'), when 'background' is set as 'yes'. However, I get the following error when 'background' is set as 'no':

Number of faces detected: 1

Yaw value mean: 8.28148317447
Looking at file: FA10_04217d118.png with model3D_aug_-00_00_01.mat
Using pose model in model3D_aug_-00_00_01.mat
Query image shape: (512, 512, 3)
Traceback (most recent call last):
File "demo.py", line 117, in
demo()
File "demo.py", line 92, in demo
model3D.ref_U, eyemask, model3D.facemask, opts)
File "/afs/crc.nd.edu/user/s/sbanerj1/vol09/WACV++/MasiFG_Data/face_specific_augm/renderer.py", line 120, in render
frontal_raw = warpImg(img, ref_U.shape[0], ref_U.shape[1], face_proj, facemask[:,0])
File "/afs/crc.nd.edu/user/s/sbanerj1/vol09/WACV++/MasiFG_Data/face_specific_augm/renderer.py", line 26, in warpImg
pixels = cv2.remap(img, np.squeeze(np.asarray(prj[0, :])).astype('float32').reshape(t_height, t_width),
ValueError: cannot reshape array of size 17789 into shape (224,224)

Irrespective of the other parameters, I always get this same error when 'background = no'. Have you experienced this before?

Thanks,
Sandipan

cv2 error in remap when running demo.py

I got the following error when run python2 ./demo.py input/input_1.jpg

Number of faces detected: 1

Yaw value mean: 1.06429849621
0 0
Looking at file: input/input_1.jpg with model3D_aug_-00_00_01.mat
Using pose model in model3D_aug_-00_00_01.mat
Query image shape: (230, 230, 3)
OpenCV(3.4.0-dev) Error: Assertion failed (dst.cols < 0x7fff && dst.rows < 0x7fff && src.cols < 0x7fff && src.rows < 0x7fff) in remap, file /home/topspin/2TB/src/opencv/modules/imgproc/src/imgwarp.cpp, line 1749
Traceback (most recent call last):
File "./demo.py", line 123, in
demo()
File "./demo.py", line 98, in demo
model3D.ref_U, eyemask, model3D.facemask, opts)
File "/home/topspin/2TB/src/face_specific_augm/renderer.py", line 109, in render
frontal_raw = warpImg(img, ref_U.shape[0], ref_U.shape[1], prj_jnt, ind_jnt)
File "/home/topspin/2TB/src/face_specific_augm/renderer.py", line 25, in warpImg
np.squeeze( np.asarray( prj[1,:] ) ).astype('float32'), cv2.INTER_CUBIC)
cv2.error: OpenCV(3.4.0-dev) /home/topspin/2TB/src/opencv/modules/imgproc/src/imgwarp.cpp:1749: error: (-215) dst.cols < 0x7fff && dst.rows < 0x7fff && src.cols < 0x7fff && src.rows < 0x7fff in function remap

too many values to unpack

when I run demo on a single image: python demo.py input.jpg
but it returns error as follows:
Traceback (most recent call last):
File "demo.py", line 117, in
demo()
File "demo.py", line 48, in demo
allModels = myutil.preload(this_path,pose_models_folder,pose_models,nSub)
File "/home/jiajiong.caojiajio/workspace/gongan_live/face_specific_augm-master/myutil.py", line 66, in preload
model3D = ThreeD_Model.FaceModel(this_path + pose_models_folder + pose, 'model3D', True)
File "/home/jiajiong.caojiajio/workspace/gongan_live/face_specific_augm-master/ThreeD_Model.py", line 15, in init
self.eyemask = self.getEyeMask(width=8,plot=False)
File "/home/jiajiong.caojiajio/workspace/gongan_live/face_specific_augm-master/ThreeD_Model.py", line 38, in getEyeMask
output1 = self.createMask(eyeLeft, width=width)
File "/home/jiajiong.caojiajio/workspace/gongan_live/face_specific_augm-master/ThreeD_Model.py", line 65, in createMask
contours, hierarchy = cv2.findContours(eyemask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
ValueError: too many values to unpack

Does anyone have any idea to figure it out?

AttributeError

Hello，I want to ask you that there is a problem that "AttributeError: 'module' object has no attribute 'pairwise_distances_argmin_min'".Probably what is the reason? Thank you！

Error:ValueError: too many values to unpack

When I run the dome error occurs:

> Preloading all the models for efficiency
> Loading pose model in model3D_aug_-00_00_01.mat
Traceback (most recent call last):
  File "demo.py", line 117, in <module>
    demo()
  File "demo.py", line 48, in demo
    allModels = myutil.preload(this_path,pose_models_folder,pose_models,nSub)
  File "/home/lch/EmotiW2018/project/face_specific_augm/myutil.py", line 66, in preload
    model3D = ThreeD_Model.FaceModel(this_path + pose_models_folder + pose, 'model3D', True)
  File "/home/lch/EmotiW2018/project/face_specific_augm/ThreeD_Model.py", line 15, in __init__
    self.eyemask = self.getEyeMask(width=8,plot=False)
  File "/home/lch/EmotiW2018/project/face_specific_augm/ThreeD_Model.py", line 38, in getEyeMask
    output1 = self.createMask(eyeLeft, width=width)
  File "/home/lch/EmotiW2018/project/face_specific_augm/ThreeD_Model.py", line 64, in createMask
    contours, hierarchy = cv2.findContours(eyemask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
ValueError: too many values to unpack

However, I can run successfully in the: https://github.com/dougsouza/face-frontalization

I want to get only the face area on the picture, and the author suggest to see your repository.
How should I fix the error?

Bad backgroung crop

Bad backgroung crop in reconstructed frontal position. Exist empty areas with background around face, especcialy then face turned left or right. How modify background substraction to make it more accurate for transfrormed face? I cheked that facial points is accurate. Why this facial points don't use for accurate background substraction?

Dlib, opencv and imutils version

Please, what is the version of Dlib, opencv and imutils ?

face expression

Does this repo have face expression augmentation? like the one in https://arxiv.org/abs/1603.07057?

How did I run the example of input1?

Hi every one, I'm a student from China.
Here's the blog of the whole process from installing opencv to run the imput1 example
May be can help you a little

http://www.cnblogs.com/QingHuan/p/6753440.html

Does the frontalization work with most image sizes, head orientations?

@iacopomasi I read that this repo is based on https://github.com/dougsouza/face-frontalization. Does it also face the same issue of poor performance with images different than test.jpg (See 1, 2, 3 or 4), or did you manage to solve such problem?

Thanks,
Dani

additional pose models

How can I create new pose models below:

model3D_aug_+22_00_01.mat to model3D_aug_+22_00_10.mat
and
model3D_aug_+44_00_01.mat to model3D_aug_+44_00_10.mat

Thanks,

Question about downloading the model

I can't download the model you provided,
and when I try to train my own caffemodel, the net accuracy only get 0.07, So what should I do to solve the problem?

Opencv 4.2.0 error

Hi,

Can you help me ?

I got this error on Opencv 4.2.0:

Traceback (most recent call last):
File "demo.py", line 117, in
demo()
File "demo.py", line 92, in demo
model3D.ref_U, eyemask, model3D.facemask, opts)
File "/home/magna/face_specific_augm/renderer.py", line 109, in render
frontal_raw = warpImg(img, ref_U.shape[0], ref_U.shape[1], prj_jnt, ind_jnt)
File "/home/magna/face_specific_augm/renderer.py", line 25, in warpImg
np.squeeze( np.asarray( prj[1,:] ) ).astype('float32'), cv2.INTER_CUBIC)
cv2.error: OpenCV(4.2.0) /io/opencv/modules/imgproc/src/imgwarp.cpp:1724: error: (-215:Assertion failed) dst.cols < SHRT_MAX && dst.rows < SHRT_MAX && src.cols < SHRT_MAX && src.rows < SHRT_MAX in function 'remap'

adding more angles and directions

Hi,

I have read https://github.com/iacopomasi/face_specific_augm/tree/master/render_mat_demo

to understand how to add more angles and directions to the face model.

it seems that each yaw angle should adjust the translation (https://github.com/iacopomasi/face_specific_augm/blob/master/render_mat_demo/adjust_translation.m), but how you determine these adjusted values?

by the way, how can I add roll and pitch?

Questions about 3D Rendering of Face Image used in Face Specific Augmentation

Hi, developers!

We are trying to create a headless renderer for facial images and we’ve seen your successful work in rendering. However, we are using different inputs for the renderer that we are trying to create. Still, we’re trying to use your project as reference. We would appreciate it if you can give us a brief explanation of how the renderer works. To be specific, what are these parameters?

• proj_matrix
• ref_U
• eyemask
• facemask
• opts

We hope to hear from you soon. Your answers can help support the computer vision research field. Thank you so much!

errors when render

hi,
when i run demo.py ,errors occur at function wrapImg in render.py
OpenCV Error: Assertion failed (dst.cols < 0x7fff && dst.rows < 0x7fff && src.cols < 0x7fff && src.rows < 0x7fff) in remap, file /home/opencv-3.4.0/modules/imgproc/src/imgwarp.cpp, line 1749
it seems that the output size of the image after remapping is out of size ,why? the input image size is 230,230,3 and the prj is 224x224x2,so,the output should be 224x224x3,right? Looking for your reply,thank you!
def warpImg(img, t_height, t_width, prj, idx):
new_img = np.zeros((t_height*t_width, 3))
## In case we have some points
if prj.size != 0:
print(len(np.squeeze( np.asarray( prj[0,:] ) ).astype('float32')))
print(len(np.asarray( prj[1,:]).T))
pixels = cv2.remap(img,np.squeeze( np.asarray( prj[0,:] ) ).astype('float32'),np.squeeze( np.asarray( prj[1,:] ) ).astype('float32'),cv2.INTER_CUBIC)
#print(pixels.shape)
pixels = pixels[:,0,:]
new_img[idx,:] = pixels

Opencv 3.2.0 errors

Hi,

I'm testing the project and I have some problems.

1. I have changed ThreeD_Model.py:63 --> contours, hierarchy = to _, contours, hierarchy =
   Because with opencv 3.2.0 findCountours return 3 params.

2. Now I can run the demo.py script, but when I have generated 10 images in output (always 10 images), I get this error:

OpenCV Error: Assertion failed (dst.cols < SHRT_MAX && dst.rows < SHRT_MAX && src.cols < SHRT_MAX && src.rows < SHRT_MAX) in remap, file <PATH_TO_OPENCV>/opencv-3.2.0/modules/imgproc/src/imgwarp.cpp, line 4956
Traceback (most recent call last):
  File "demo.py", line 111, in <module>
    demo()
  File "demo.py", line 86, in demo
    model3D.ref_U, eyemask, model3D.facemask, opts)
  File "<PATH_TO_FACESPECIFICAUGM>/faceSpecificAugm/renderer.py", line 109, in render
    frontal_raw = warpImg(img, ref_U.shape[0], ref_U.shape[1], prj_jnt, ind_jnt)
  File "<PATH_TO_FACESPECIFICAUGM>/faceSpecificAugm/renderer.py", line 25, in warpImg
    np.squeeze( np.asarray( prj[1,:] ) ).astype('float32'),  cv2.INTER_CUBIC)
cv2.error: <PATH_TO_OPENCV>/opencv-3.2.0/modules/imgproc/src/imgwarp.cpp:4956: error: (-215) dst.cols < SHRT_MAX && dst.rows < SHRT_MAX && src.cols < SHRT_MAX && src.rows < SHRT_MAX in function remap

Any ideas? Thanks!!!

Rendering of other objects

Is this code extensible to the rendering of other objects (such as vehicles), assuming that the face landmark detection step can be modified?

Demo might be broken

I just committed #8 and was wondering which output the cmd:
python demo.py image.jpg
should be, can you please explain? image.jpg is a simple image file containing a face. The script ran and nothing happens.

when i run the demo, there occured an error in function warpImg, can anyone help? thx

Traceback (most recent call last):
File "demo.py", line 117, in
demo()
File "demo.py", line 92, in demo
model3D.ref_U, eyemask, model3D.facemask, opts)
File "/home/joey/Downloads/pycharm_projects/face_specific_augm-master/renderer.py", line 109, in render
frontal_raw = warpImg(img, ref_U.shape[0], ref_U.shape[1], prj_jnt, ind_jnt)
File "/home/joey/Downloads/pycharm_projects/face_specific_augm-master/renderer.py", line 25, in warpImg
np.squeeze( np.asarray( prj[1,:] ) ).astype('float32'), cv2.INTER_CUBIC)
cv2.error: OpenCV(3.4.1) /io/opencv/modules/imgproc/src/imgwarp.cpp:1749: error: (-215) dst.cols < 32767 && dst.rows < 32767 && src.cols < 32767 && src.rows < 32767 in function remap

3D models

I have sent an email to you, hope you can send me a copy of them to generate my own views

Request

Thank you very much for the code!

A quick question: is it possible to get a full resolution frontalized face, or is it constrained to the size of the CNN input (e.g., 160)? In other words, if I have a source image that contains a 500x500 pixel face, is it always going to be downsized to 160 x 160 after frontalization?

Thank you,
MS

how to add other pose of 3dmodels

hello, how do you create the 3d models and whats the difference of the 10 different models in one pose。Besides,what the 3d models inside contains and how can i create new pose model?
(I am a new beginner and a student,thank you very much.)

Error

It was working fine for a minute or so... and then suddenly this error appears when I try to render a new image

Number of faces detected: 1
('> Yaw value mean: ', 0.093203850386659223)

Looking at file: input/input_1.jpg with model3D_aug_-75_00_01.mat
Using pose model in model3D_aug_-75_00_01.mat
('> Query image shape:', (230L, 230L, 3L))
skipping sym
Traceback (most recent call last):
File "demo.py", line 117, in
demo()
File "demo.py", line 98, in demo
rendered_raw = myutil.cropFunc(pose,rendered_raw,crop_models[poseId])
IndexError: list index out of range

ThreeD_Model.py line 62 error

when run demo.py test.jpg
error:
face_specific_augm-master/ThreeD_Model.py", line 62, in createMask contours, hierarchy = cv2.findContours(eyemask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)ValueError: too many values to unpack
thx~