jhb86253817/pipnet

Efficient facial landmark detector

pipnet's Issues

train_gssl wrong

File "/home/inspur/Projects/PIPNet/lib/functions_gssl.py", line 85, in compute_loss_pip │ outputs_local_x_select = torch.gather(outputs_local_x, 1, labels_max_ids) │ RuntimeError: index 18 is out of bounds for dimension 1 with size 16
when I use train_gssl.py, the problem comes. Below is my config
`# with curriculum

class Config():
def init(self):
self.det_head = 'pip'
self.net_stride = 32
self.batch_size = 16
self.init_lr = 0.0001
self.num_epochs = 60
self.decay_steps = [30, 50]
self.input_size = 256
self.backbone = 'resnet18'
self.pretrained = True
self.criterion_cls = 'l2'
self.criterion_reg = 'l1'
self.cls_loss_weight = 10
self.reg_loss_weight = 1
self.num_lms = 68
self.save_interval = self.num_epochs
self.num_nb = 10
self.use_gpu = False
self.gpu_id = 1
self.curriculum = True`

How to train my own dataset?

hey,thanks to your contritubution for keypoints detection! I wanna know how to train my own datasets? I just wanna detect 2 keypoints,instead of 68 keypoints.

Questions about the confidence of key points

Hi,I'm sorry, I still don't understand,
the max_cls returned by function forward_pip is the confidence scores of predicted landmarks.
why is the confidence of the occluded points higher than that of the unoccluded points?
For example, if the face is partially occluded, I want to know which key points are occluded，Or the occluded key points are not displayed.
Can it be achieved based on the confidence of key points?

why target *(input/netstride)?

i confuse about : mu_x = int(floor(target[i][0] * map_width)),why target[i][0] * map_width not target[i][0]

The predicted landmarks appear out of face box

In order to shorten the detection time of the face detect, I predict the face box of the current frame by using the landmark of the previous frame of the image. In some cases, the landmarks predicted by PIPNet predicted would appear outside my face box.

In the image below, the area in the red box is the image I entered into the network, and the green dot is the landmarks predicted by PIPNet.

Some of these landmarks appear outside the box, does anyone know why?

Training only the eyes and mouth parts of the key points, there is a strange problem

Hi，I only use the eyes and mouth parts of the key point to train PIPNET, but there is a strange problem.
I used the same model to predict key points in two videos, one of which predicted eye and mouth key points normally, while the other predicted key points at the edge of the face

Is the stride or feature map size influence the NME?

I trained on my dataset and use the input size 64X80, and found the NME is larger than the regression method before used, and when I modified the input size to 256X256, NME is lower than before. @jhb86253817

RuntimeError: _th_floor_out not supported on CUDAType for Long

HI
I am trying to test the model through sh demo.sh.
However, when I run it, I get the following error:

File "/home/main/project/PIPNet/lib/functions.py", line 185, in forward_pip
tmp_y = (torch.floor(max_ids/tmp_width)).view(-1,1).float()+outputs_y_select.view(-1,1).float()
RuntimeError: _th_floor_out not supported on CUDAType for Long

I am wondering if it is a version error.
My torch version is 1.4.

Thank you.

get_meanface error

Hi, when I run meanface_indices, _, _, _ = get_meanface(os.path.join('../data', 'data_300W', 'meanface.txt'), 68) it reported indexError:

File "/work/landmark/PIPNet/lib/data_utils.py", line 170, in
meanface_indices, _, _, _ = get_meanface(os.path.join('../data', 'data_300W', 'meanface.txt'), 68)
File "/work/landmark/PIPNet/lib/functions.py", line 53, in get_meanface
meanface_indices_reversed[meanface_indices[i][j]][0].append(i)
IndexError: index 67 is out of bounds for axis 0 with size 67

troubleshooting sh run_train.sh

Hi!

Thank you very much for this and making it public!

I have tried running the training for ResNet101 for 300W dataset (I am using Python on WSL with Ubuntu distro) and I have been getting this error.

"Traceback (most recent call last):
File "lib/train.py", line 32, in
my_config = importlib.import_module(config_path, package = 'PIPNet')
File "/home/apecundo725/anaconda3/envs/PIPNet/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 970, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'PIPNet.experiments.data_300W.pip_32_16_60_r101_l2_l1_10_1_nb10.'; 'PIPNet.experiments.data_300W.pip_32_16_60_r101_l2_l1_10_1_nb10' is not a package"

I have kept the folder structure as is from the github but I keep getting this error. I am wondering what fixes can be applied (I have tried adding init.py files but it does not help)

I have been able to run the preprocessing.py script but am stuck at the training part of the process.

Thank you very much for the support.

About some variable

PIPNet/experiments/WFLW/pip_32_16_60_r101_l2_l1_10_1_nb10.py

Line 19 in 77e01c8

self.num_nb = 10

What is the num_nb ?

could you please share WFLW-68 annotations

Official WFLW only provide 98 landmarks annotation, could you please give me someway to download the WFLW-68 annotations, or share your code about generate that? Thank u so much!

Training with data where some points are missing from GT

Hello, thank you for your work. Is it possible to train the model with additional custom data, where not all the points have Ground Truth positions? For example, I want to train for 68 landmarks, but several images from my custom data are only annotated with 40 points and the remaining 28 points are not annotated.

About LaPa preprocess.py part

Hello, I was working on the LaPa dataset part of your landmark detection model. You shared the pretrained model of LaPa dataset but preprocess part is empty which is important for training. I know that preprocess is cropping the faces from dataset images and transforming them into 256x256 size. However, LaPa dataset does not contain face bounding coordinates unlike other datasets. How did you train your model without cropping images? Or did you manually select the face box coordinates?

when running "sh make.sh" in windows I get this error

error D8021
error: command 'C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.30.30705\bin\HostX86\x64\cl.exe' failed with exit code 2

Is FaceBoxesV2 compatible with windows or am I doing something wrong?.

Thanks in advance.

excellent work

Very practical work! It helps a lot!

Traing & Test

Is the current code ready to train now?

some question when general the labels for train

Hi, it is very nice work! I get some question when general the labels for train. In the paper, only one point in the score map is set 1, others are set 0, and one point in the offset map is set offset,others are set 0. In the inference phrase, how to promise all the score maps predicted are right? if there is one score map predicted is wrong, which could lead a wrong index for offset map in testing, the final landmark coordinate couldn't be right! why not set all possible offset in the gird?

increasing accuracy if using only eye eyebrow nose landmark

Hi, thanks for great works.
I have some question, if only eye, eyebrow, nose part for training, accuracy is increasing? my concern is large pose and occlusion image make some problem like lower performance even if using resnet101. Do u have some idea? If I using multiple dataset, will it be good?

thanks

crop process issue

PIPNet/lib/demo_video.py

Line 101 in b9eab58

det_ymin += int(det_height * (det_box_scale-1)/2)

Should line101 be changed to:
det_ymin -= int(det_height * (det_box_scale - 1) / 2)

same in demo.py

Problem about UDA training with curriculum

when I tried to train with unlabeled data, I got results like this:

I followed your guide in README.md and run
python lib/train_gssl.py experiments/data_300W_COFW_WFLW/pip_32_16_60_r18_l2_l1_10_1_nb10_wcc.py

I had no idea what the problem was.

How to detect abnormal landmark prediction?

For inference, we still have some image that have landmark prediction very bad which has wrong face shape (compare with regression method)? Could you suggest any way to filter the result ? Note that, I am using video as input and landmark score that we used may not good

Can this project include the forehead?

Excuse me,How to get the data for this file,'meanface.txt'

My understanding is that this line of code is to get the index of the 10 coordinate points of the NMS.
But where did this initial file（,'meanface.txt'） come from?
Please point out if I understand wrong, thank you！
meanface_indices, reverse_index1, reverse_index2, max_len = get_meanface(os.path.join(rootpath,'data', cfg.data_name, 'meanface.txt'), cfg.num_nb)

how could i change the 98 keypoints to 68 keypoints model

could i change to 68 keypoints model, or is 68 facial keypoint data somehow related to 98key point?

Question about training parameters

I'm trying to use the PIPNet framework to train a custom model unrelated to human faces with a non-standard number of keypoints (4). I'm confused about this line in lib/train.py:

labels = get_label(cfg.data_name, 'train.txt')

what is labels, what format is it and what is train.txt?

Where is celeba_bboxes.txt?

I download all files from CelebA, but I do not find celeba_bboxes.txt. So I want to ask you where it is. Thank you very much.

Train custom dataset.

Hello, author thanks for sharing your code.

i want to training my custom dataset, but when i run train.py, i have an Error named 'ModuleNotFoundError: No module named 'PIPNet' '. so how can i get PIPNet module?

gssl training

Hello, I would like to consult the relevant details of gssl training. May I ask whether a net is initialized for each task when using gssl training in the paper, and is the latest training net predicted for each pseudo-label?

some question about curriculum training

Hello, it is very nice work! I have some question about curriculum training.
I have attention that the line 239-242 in train_gssl.py，that mean every loop in task_type_list you should build a new network model in the code ? And I have no find current loop training loading last loop saved weight?
this way can promise the model learning easy to hard?
Looking forward to your reply.

Very low GPU utilizaiton ratio

Hello,

thank you very much for the nice work.

I found the GPU utilization ratio is very low, around 10%.

Could you please kindly help and tell how to speed up landmark detection?

Thank you!

Can the project be deployed on WINDOWS

Can the project be deployed on WINDOWS？

PIPNets

hello, the url((https://drive.google.com/drive/folders/17OwDgJUfuc5_ymQ3QruD8pUnh5zHreP2?usp=sharing) cann't download, thanks!

AFLW

tf2 model or tflite

Hi, do u have tf model or tflite model?

BarrelNet code release

Hi are you planning to release your newer code for BarrelNet?

How to train on my own dataset?

Hello， I have a batch of expanded images and corresponding ldmk annotations. I want the model to be trained both on 300W and my own data at the same time. How should I do?

About test Image

Hello, author, thanks for your sharing.

Can this network used to test CelebA-HQ-img? I want found a network to calculate landmark loss in CelebA-HQ-img, but after I seen your code, I don't konw how to test in CelebA-HQ-img.

snapshot cannot download

Hello, the url(https://drive.google.com/drive/folders/1WruSmUzb2mqZnSiqWO8wRWZcttaxmNnZ) of snapshot is no longer accessible.

from .nms.cpu_nms import cpu_nms, cpu_soft_nms ModuleNotFoundError: No module named 'utils.nms.cpu_nms'

from .nms.cpu_nms import cpu_nms, cpu_soft_nms ModuleNotFoundError: No module named 'utils.nms.cpu_nms'
I've run make.sh, but still get this error

Only 19 of 21 landmarks in AFLW use? Why? Can not found settings in CPM.

hi~ I have found that you only use 19 of 21 landmarks in AFLW, and point out that it's follow CPW. But, I can't found any information about AFLW setting in CPM's paper. Could you please release the detail indices of 19 landmarks you have used in you paper?

I see the code in train.py:

points_flip = [6, 5, 4, 3, 2, 1, 12, 11, 10, 9, 8, 7, 15, 14, 13, 18, 17, 16, 19]  # no 0 and 20
points_flip = (np.array(points_flip)-1).tolist()  # then from 0~18, total 19 points

so, it means that you have exclude 0 and 20 in AFLW ?

How to get 98 facial keypoints model?

How can I get your pre-trained 98 key points model?
which is showed in run_demo.sh （r18）

Cfg file for running the demo of MobileNetv2 and MobileNetV3 models

First of all, thanks for sharing the excellent models. I've tested the resnet101 based models, it works perfect. And I noticed that you guys also provide the mobilenet-v3-large model (mobilenetv3-large-1cd25616.pth) in ~/PIPNet/lib.

I'd like to test the effect of this model, but can't find the related cfg file, seems that only Resnet related cfg files have been provided.
Could you provide the 'cfg' file and also the 'data_name' which may needed in calculating the get_meanface function.

How to get the credibility of the key points of the face？

Some parts of the face may be occluded. The key points should be predicted. Can you get the credibility value?

relative import error

Cannot change the parameters in config

Hi, this is my first time asking questions on GitHub, please point out if anything is unclear

After changing the parameters in Config(), I cannot change them again (num_nb). My config is shown below:
class Config(): def __init__(self): self.det_head = 'pip' self.net_stride = 32 self.batch_size = 16 self.init_lr = 0.0001 self.num_epochs = 60 self.decay_steps = [30, 50] self.input_size = 256 self.backbone = 'resnet18' self.pretrained = True self.criterion_cls = 'l2' self.criterion_reg = 'l1' self.cls_loss_weight = 10 self.reg_loss_weight = 1 self.num_lms = 4 self.save_interval = self.num_epochs self.num_nb = 0 self.use_gpu = True self.gpu_id = 0

But when running run_train.sh, it showed the last parameter.

experiment_name: pip_32_16_60_r18_l2_l1_10_1_nb10 data_name: CFP det_head: pip net_stride: 32 batch_size: 16 init_lr: 0.0001 num_epochs: 60 decay_steps: [30, 50] input_size: 256 backbone: resnet18 pretrained: True criterion_cls: l2 criterion_reg: l1 cls_loss_weight: 10 reg_loss_weight: 1 num_lms: 4 save_interval: 60 _num_nb: 10_ use_gpu: True gpu_id: 2
I was confused for a long time,so I came to ask you.

train_gssl.py problem

Hello, when I use train_gssl.py, "300W nme is inf", is this normal?

300W nme: inf
COFW nme: 0.05221280819123009
WFLW nme: 0.08741295886838747

Eyes keypoints accurate

Hello, sorry to borther you again. I train the resnet18 with "data_300W_CELEBA/pip_32_16_60_r18_l2_l1_10_1_nb10_wcc.py" , when the eyes are open, the result is good as your model, but when eyes are closed, the eyes keypoints are bad. Are there some tricks to improve the accurate of eyes keypoints

sh make.sh

这个运行完之后，没有出现文件的转

换是什么原因，请帮助我

Reproducing NME on test data

Hi
Thank you for sharing your codes.

I have downloaded one of the dataset (COFW) and tested using available pre-trained models (resnet18). The NME score that I got is 0.08758. If I multiply the score with 100%, it results in 8.758%.

Q: How to produce the same NME score as your paper? In the COFW, it is reported as 3.31%. Thank you.