github-luffy/pfld_68points_pytorch

Implementation of PFLD For 68 Facial Landmarks By Pytorch

PFLD_68Points_Pytorch

Implementation of PFLD For 68 Facial Landmarks By Pytorch

DataSets

• WFLW Dataset

  Wider Facial Landmarks in-the-wild (WFLW) is a new proposed face dataset. It contains 10000 faces (7500 for training and 2500 for testing) with 98 fully manual annotated landmarks.

  1.Training and Testing images[Google Drive][Baidu Drive], Unzip and put to ./data/WFLW/raw/

  2.Have got list_68pt_rect_attr_train.txt and list_68pt_rect_attr_test.txt. If you want to get them by youself, please watch get68psFrom98psWFLW.py and run it before please get WFLW Face Annotations , unzip and put to ./data/WFLW/

  3.Move Mirror68.txt to ./data/WFLW/annotations/

   $ cd ./data/WFLW 
   $ python3 WFLW_SetPreparation68.py

• 300W Dataset

  300W is a very general face alignment dataset. It has a total of 3148+689 images, each image contains more than one face, but only one face is labeled for each image.File directory includes afw(337)，helen(train 2000+test 330)，ibug(135)，lfpw(train 811+test 224) with 68 fully manual annotated landmarks.

  1.Training and Testing images[Databases][Baidu Drive], Unzip and put to ./data/300W/raw/

  2.Have got list_68pt_rect_attr_train.txt and list_68pt_rect_attr_test.txt. If you want to get them by youself, please watch get68pointsfor300W.py and run it

  3.Move Mirror68.txt to ./data/300W/annotations/

   $ cd ./data/300W 
   $ python3 300W_SetPreparation68.py

• 300VW Dataset

  300VW is a video format, which needs to be processed into a single frame picture and corresponds to each key point pts file.

  1.Training and Testing images[Databases], Unzip and put to ./data/300VW/raw/

  2.Run get68psAndImagesFrom300VW.py to get list_68pt_rect_attr_train.txt

  3.Move Mirror68.txt to ./data/300VW/annotations/

   $ cd ./data/300VW 
   $ python3 get68psAndImagesFrom300VW.py
   $ python3 300VW_SetPreparation68.py

• Your Own Dataset

  If you want to get facial landmarks for new face data, please use Detect API of face++. For specific operations,
  please refer to API Document. And refer to ./data/getNewFacialLandmarksFromFacePP.py for using the api interface.

• All Dataset

  After completing the steps of each data set above, you can run the code merge_files.py directly .

   $ cd ./data
   $ python3 merge_files.py

training & testing

training :

 $ sh train.sh

reading images from a camera to test:

 $ python3 camera.py

reading images from a dir to test:

 $ python3 test.py

Result

Sample IMGs:

Details about the models are below:

tip: please install resnest to use ResNest models.

Name	# Params	Mean error	Failure rate	One iteration time(s)
ResNest50	122.27M	0.046	0.038	0.304
MobileNetV2_0.25	1.09M	0.075	0.174	0.154
MobileNetV2_1.00	7.28M	0.065	0.127	0.203
BlazeLandmark	7.52M	0.069	0.131	0.171
HRNetV2	545.07M	0.066	0.125	0.769
efficientnet-b0	16.67M	0.064	0.119	0.202
efficientnet-b1	26.37M	0.075	0.149	0.252
efficientnet-b2	30.85M	0.071	0.145	0.266
efficientnet-b3	42.29M	0.099	0.136	0.307
efficientnet-b4	68.34M	0.077	0.164	0.375
efficientnet-b5	109.34M	0.094	0.173	0.501
efficientnet-b6	156.34M	0.081	0.175	0.702
efficientnet-b7	244.03M	0.081	0.196	0.914

pytorch -> onnx -> ncnn

Pytorch -> onnx -> onnx_sim

Make sure pip3 install onnx-simplifier

 $ python3 pytorch2onnx.py
 $ python3 -m onnxsim model.onnx model_sim.onnx

onnx_sim -> ncnn

How to build :https://github.com/Tencent/ncnn/wiki/how-to-build

 $ cd ncnn/build/tools/onnx
 $ ./onnx2ncnn model_sim.onnx model_sim.param model_sim.bin

reference:

PFLD: A Practical Facial Landmark Detector https://arxiv.org/pdf/1902.10859.pdf

ResNest: Split-Attention Networks https://hangzhang.org/files/resnest.pdf

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks https://arxiv.org/pdf/1905.11946.pdf

pytorch：https://github.com/lukemelas/EfficientNet-PyTorch

tensorflow：https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet

keras: https://github.com/qubvel/efficientnet

Tensorflow Implementation for 98 Facial Landmarks: https://github.com/guoqiangqi/PFLD

pfld_68points_pytorch's Issues

Loss saturate after 80 Epoch

Hi @github-luffy,
I trained AuxiliaryNet on WFLW data set using WingLoss and loss and failure rate: L1 staturate after 80 Epoch. Does this mean model is converged/generalised?
What was your final loss during your training?

Epoch:[101  ][100 /293 ]	Loss: 12.227	 lr 1e-05	 average_time:0.269s	 remain_time:20.540h
Epoch:[101  ][200 /293 ]	Loss: 12.547	 lr 1e-05	 average_time:0.269s	 remain_time:20.520h
Epoch:[101  ][293 /293 ]	Loss: 11.627	 lr 1e-05	 average_time:0.269s	 remain_time:20.500h
Test epochs: 10	Loss 0.102
mean error and failure rate
mean error : 0.108
failure rate: L1 0.397

检测速度

为什么论文中有毫秒级别的检测速度，但是表格中只有0.2s左右

人脸质量筛选

没有人脸，和人脸偏的很厉害的情形，反而会预测出类似正脸的结果，PFLD没有处理策略。这样会影响人脸质量筛选，无法筛选出轨迹中的好脸，大佬有没有研究过这个问题，有没有什么解决策略？

torch.nn.modules.module.ModuleAttributeError: 'MobileNetV2' object has no attribute 'module'

This happens when try to use test.py to load the model:

pfld_backbone.load_state_dict(torch.load(args.model_path, map_location= device))

To fix this issue,
we have to change save model in the "train_model.py"

---

save model

checkpoint_path = os.path.join(model_dir, 'model_'+str(epoch)+'.pth')
if args.all_model:
torch.save(model, checkpoint_path)

as
torch.save(model.state_dict(), checkpoint_path)

after that all works.

I am also wondering about an alternative without changing the train_model.py. So instead of using the train_model.py, we could do the code like below:

model = torch.load(pretrained_model)
test(test_loader, model, args, device)
...

ModuleNotFoundError: No module named 'tensorflow'

大佬你好，这个代码还是需要安装tensorflow是吗？

模型

请问预训有大模型吗 resnet那种

300W百度网盘的提取密码是？

NO config information of MobilenetV2

The code has no config information of MobilenetV2:
`class MobileNetV2(nn.Module):

def __init__(self, input_channels=3, num_of_channels=None, nums_class=136, activation=nn.ReLU6):
    super(MobileNetV2, self).__init__()
    assert num_of_channels is not None
    self.num_of_channels = num_of_channels
    self.conv1 = nn.Conv2d(input_channels, self.num_of_channels[0], kernel_size=3, stride=2, padding=1)
    self.bn1 = nn.BatchNorm2d(self.num_of_channels[0])

    self.depth_conv2 = nn.Conv2d(self.num_of_channels[0], self.num_of_channels[0], kernel_size=3, stride=1,
                                 padding=1, groups=self.num_of_channels[0])
    self.bn2 = nn.BatchNorm2d(self.num_of_channels[0])`

Can you provide the imformation?

欧拉角这部分怎么理解的呢

欧拉角那块代码没看明白

有没有使用PFLD做头部姿态估计的代码，而不是使用dlib

本人小白 自己想改一改代码，发现有点难，希望可以指导一下 使用PFLD进行头部姿态估计

脸离摄像头较近关键点出现较大偏移

脸离摄像头较近（脸占据照片的绝大部分）关键点会出现较大偏移，问题可能出现在哪里？如何解决？谢谢

可视化标注数据

@github-luffy
您好，请教一下关于人脸关键点的标注信息
我可视化了一下list.txt的关键点，发现部分点是乱的，请问标注的点是直接可视化么？
我是读取一行，然后line[1]~line[136] *112变成68个点画在图上的，另外标注信息最后的0 0 1 0 0 0 是什么？

/home/data/WFLW/train_data/imgs/WFLW_0_51_Dresses_wearingdress_51_377_0.png 0.0885083825019613 0.20997702626503661 0.1156312312042364 0.30337741484701886 0.09992738348669587 0.3996145336199006 0.10177439925062108 0.4969836119328583 0.13573235547692208 0.5884961243952668 0.17954538257550995 0.6759568218406773 0.22458856175634154 0.7627766501454629 0.26553019120603427 0.8514915051320606 0.34426656427742547 0.905014995750523 0.47679674475761635 0.887883525513206 0.6020233281985486 0.8372941914961428 0.720325805153308 0.7726321679278897 0.8104811153651281 0.6731986201457898 0.8613277419341658 0.5485699386277458 0.8872600220237317 0.41602928768142 0.9057656770969534 0.28222145495554396 0.9173093660107218 0.1476206759528635 0.0840376849952602 0.14829704252745815 0.10764852228523797 0.10126160178723195 0.1347259138418541 0.11636607916285303 0.16502725130344534 0.13154180678363625 0.1992426916146378 0.15204183027834076 0.33944351004756146 0.13965060820639383 0.4136192928298248 0.11499374581181354 0.4939853556485356 0.09573019498561715 0.5717427401363101 0.09954604144874477 0.6633368456214043 0.14362759769711037 0.2843995672888337 0.2612719675487055 0.27263165717344406 0.34170953598980125 0.2638164823524124 0.42215387192730125 0.24054912742710513 0.49059496763859833 0.2069163302497385 0.5214749260427562 0.24675530469567208 0.5337558889987578 0.288239802276739 0.5332414315834205 0.3367609000106237 0.5220229655628923 0.385994436351824 0.5147365426418672 0.11486612104471758 0.2780711920191553 0.1281346915656054 0.2543223871845581 0.18886026677726203 0.25213903961820083 0.23631720762372516 0.2737046245750523 0.18988560632681747 0.28335252067533995 0.12916003111516083 0.2855358682416972 0.40344557502778505 0.26955653234505755 0.43627706232430047 0.24696739547921026 0.5210153747303217 0.2383965528160957 0.5729221998398274 0.25241471933021703 0.5243087193955936 0.2688514358328975 0.43957040698957245 0.27742240618462344 0.21814118070083682 0.6402741116958682 0.22741335306207505 0.6058980391115324 0.2642363225067011 0.6044979334875131 0.2938455238501896 0.6062975767766082 0.3308172505270986 0.6009817721953452 0.4262184558054393 0.6068850720776674 0.5220561646018567 0.6084555143092966 0.4513672385754446 0.6566068896688677 0.3733126588446326 0.6908674040598849 0.2898784940711624 0.7071479573908211 0.2553494505303674 0.6986364899320083 0.23048732869295893 0.6738099932171809 0.22610039491533734 0.6411255393566946 0.26013675194903896 0.645261756545829 0.294367004139154 0.6469429048035434 0.3939403071064331 0.6364537961313416 0.4926116177227707 0.6193356055096103 0.393806936351824 0.6364710340938807 0.2940980919235421 0.6469460970188284 0.2600028704399843 0.6452489876846889 0 0 1 0 0 0

AuxiliaryNet的优化问题

在train_model.py 中包含AuxiliaryNet的实例化，转 train()模式，但是在优化器，及整个过程中并没有对AuxiliaryNet参数更新的过程。 请教该如何理解AuxiliaryNet在模型训练中的作用？

Environment setup?

Any requirements.txt for development environment?

loss or NME computing for convergence

I have tried both with "pretrained_model/mobileNetV2_0.25.pth" and without this pretrained model for retraining the model for common, challenge and fullset.

However, losses never goes down to below 30, neither from pretrained or retrained from initial-scratch.
On the other hand, I tried to removed the training part of the code from train_model.py
see this
https://github.com/epoc88/PFLD_68pts_Pytorch/blob/master/test_NME.py

I did not add any pretrained model, but I still got some NME result..

Test epochs: 3 Loss 0.094
mean error and failure rate
mean error : 0.061
failure rate: L1 0.161

So I am a bit confused...

The issue could be related to optimizer and gradient decent., or loss function WingLoss is good for fine tuning, but PFLF's MSE could be good in the beginning.

Here is a Chinese paper discussing the performance of PFLD on 300W, especially the loss function
https://jishuin.proginn.com/p/763bfbd29621

One more thing, it is worth to mention that the default code for training is "common subset" for validation during training (i.e. without ibug images). There are common, challenge, fullset as validation subset.

---

我已经尝试了使用“pretrained_model/mobileNetV2_0.25.pth”和没有此预训练模型的重新训练模型，以进行普通、挑战和全集的重新训练。

然而，损失从不低于30 ，无论是从最初的训练或再训练。
另一方面，我试图从train_model.py中删除代码的训练部分
看看这个
https://github.com/epoc88/PFLD_68pts_Pytorch/blob/master/test_NME.py

我没有添加任何预先训练的模型，但我仍然得到了一些NME的结果。

测试阶段：3损耗0.094
平均误差和故障率
平均误差： 0.061
失效率：L1 0.161

所以我有点困惑...

这个问题可能与损耗函数 和 梯度下降法有关。**

下文是一篇关于300W的PFLD性能，特别是损耗函数的中文论文。
https://jishuin.proginn.com/p/763bfbd29621

这是我用您的代码修改后训练的MobileNetV2_0.25模型来测算的结果

论文结果	Speed on CPU (1.25ms)	Model Size (2MB)	NME (ION) Common (3.03)	NME (ION) Challenge (5.15 )	NME (ION) Fullset (3.45)	AUC (0.80)
我们代码的结果,	206	1.1	4.74	7.95	5.25	0.4889 (fullset)

主要差别，论文速度1.25ms, 我们的是200ms, MobileV2_0.25 的尺寸也不一样，论文是2.1MB， 我们pretrained_model 是1.1MB, 这个不是官方的pretrained model?

还有一点，值得一提的是，训练的默认代码是训练期间验证的“公共子集”（即。没有ibug图片）。有共同的，挑战的，全集作为验证子集。

还想没有300W的困难数据的txt文件（ibug的130多张图的）

train_model.py中的一点疑问

hi，我在改代码的时候发现，你的train_model.py 里 line158 是这样的：

pre_landmarks, auxiliary_features = model(images_batch)

而 line201 里是这样的：

pre_landmarks, euler_angles_pre = model(images_batch)

line201 是写错了吗？

没有依赖包版本文件

作者你好，我要在30显卡上跑，cuda得11以上，其他的python包需要什么版本的呢？

模型转换部署

onnx模型如何转成tflite模型？

眼部单独训练

您好，如果我需要提高眼部精度，只训练眼部12个关键点，数据预处理增强是否需要变动?训练数据应该是完整人脸还是以眼睛为中心的部分脸?欧拉角那里应该怎么处理？

代码loss中没有考虑类不平衡问题的属性权重？

你好， 代码loss中没有考虑类不平衡问题的属性权重？

您好，您这个代码不是使用的PFLD的PFLDLoss而是wing_loss？所以这个和PFLD方法的关系是什么？

300w数据集的百度云密码是多少

请问有resnet50检测人脸关键点的代码吗？

感谢分享，请问能提供resnet50训练并测试人脸关键点的代码吗？非常感谢！

关于为什么要用Auxiliary net的问题

论文中有这样一段描述：
One may wonder that given predicted and ground-truth
landmarks, why not directly compute the Euler angles from
them? Technically, it is feasible. However, the landmark
prediction may be too inaccurate especially at the beginning
of training, which consequently results in a low-quality estimation of the angles. This could drag the training into
dilemmas, like over-penalization and slow convergence. To
decouple the estimation of rotation information from landmark localization, we bring the auxiliary subnet.
这里只解释了为什么不能用训练得到的landmarks去计算Eular角，但为什么不能直接用标注数据去计算Eular角呢？（用本身的标注数据去计算Eular角不会更准确吗？）

读取数据出错

你好作者，请问在读取数据训练时出错是怎么回事

Traceback (most recent call last):
File "train_model.py", line 307, in
main(parse_arguments(sys.argv[1:]))
File "train_model.py", line 63, in main
is_train=True)
File "/home/ctgcdt/zhulei/PFLD_68points_Pytorch-master/generate_data.py", line 14, in init
self.file_list, self.landmarks, self.attributes = gen_data(file_list)
File "/home/ctgcdt/zhulei/PFLD_68points_Pytorch-master/generate_data.py", line 79, in gen_data
attribute = np.asarray(line[137:], dtype=np.int32)
ValueError: invalid literal for int() with base 10: 'lfpw/trainset/image_0154.png'

正常最终的损失值大概为多少？

您好，我想请教下，我直接跑demo，用的是efficient b0，loss下降到30左右就降不下去了，不知这是不是正常的情况？

the download address of Baidu cloud for300W Dataset

关于PFLD网络的精度

大佬，有个问题请教下，PFLD这个算法的精度怎么样，大佬你实现的这个版本精度有达到论文里说的水平不，纯新手小白一枚，想找个近两年最新最好的网络学习一下。

代码的数据处理中没看到crop部分

看到生成训练数据的时候，根据关键点点位信息生成了人脸框信息，但是在dataloader里似乎是直接读取图片地址，未进行crop操作。是因为原始训练数据都是crop之后的嘛？

Does model also provides each landmark confidence score

@github-luffy Does models also provides each landmark confidence score? if no then have any idea how we can get from network?

300W的setpreparation部分

你好！我在运行300wsetpreparation.py时出现了一些错误：

/project/PFLD_N/data/300W/train_data
3283
Traceback (most recent call last):
File "300W_SetPreparation68.py", line 226, in
imgs = get_dataset_list(imgDir, outDir, landmarkDir, is_train)
File "300W_SetPreparation68.py", line 194, in get_dataset_list
Img.load_data(is_train, 10, Mirror_file)
File "300W_SetPreparation68.py", line 87, in load_data
height, width, _ = img.shape
AttributeError: 'NoneType' object has no attribute 'shape'

但是检查后都是按照readme的规定来做的，而且WFLW的set.py可以正确运行，请问是哪里出现了问题吗？

"getNewFacialLandmarksFromFacePP.py" is incomplete

"getNewFacialLandmarksFromFacePP.py" is incomplete,Can you provide the full version?

BlazeLandMark预训练模型

感谢luffy的分享，想测试下BlazeLandMark模型的效果，请问能提供一下预训练模型吗

camera.py运行自动闪退了

运行 camera.py 自动会闪退
AttributeError: 'BlazeLandMark' object has no attribute 'secondconv'
无论是用 视频文件还是接摄像头都会闪退
这里我的权重文件是用项目中提供的 model_10.pth

关于coco里的关键点

@github-luffy
你好 请问一下为何PFLD都没有用coco里的68个face关键点呢？

请问你这边提供的预训练模型是使用的什么数据集训练的？

CPU上的测试时间？

你好，cpu上的测试时间能达到论文中所说的速度吗

有关3D人脸坐标的问题

您好作者，感谢你在前段时间不辞辛劳的回复，还有个问题想请教下，就是像98、106这样的特征点对应的标准3D模型坐标怎么获取呢，您有经验吗，谢谢指教

关于使用300W数据集进行训练的问题

作者您好！首先谢谢您的工作！
我想问下既然在300W数据集中没有关于6种人脸属性的描述，为什么不在数据准备、训练过程中把这部分去掉呢？

请问ResNest50模型结果是如何训练得到的呀？

这是个很棒的项目~
我看到给出的结果中，ResNest50的mean error=0.046，failure rate=0.038
我用WFLW数据集跑了100个epoch，最后分数是 mean error=0.086,failure rate=0.241
请问是不是因为训练轮数太短，或者是需要一些其它训练技巧，亦或使用的数据集存在差异？

300W Dataset baiduyun 验证码是?

百度云提取码可以提供下么？

没有找到百度云提取码，可否提供下，非常感谢~

训练多少个epoch后loss能正常收敛

你好作者！
我训练efficientnet-b0，第三十个epoch了loss还非常大，这正常吗？
Epoch:[34 ][26 /26 ] Loss: 29673.287 lr 1e-05 average_time:3.871s remain_time:13.001h
Test epochs: 5 Loss 46748736.800
mean error and failure rate
mean error : 1.672
failure rate: L1 0.866

What's the final loss ?

What is the final loss you get? I tried to reimplement PFLD using keras. When I was training, the loss saturated around 30 at the early beginning and the loss was still around 30 after 20k iterations.

test代码的算error分母好像不对

分母应该是样本的个数，但是代码写的是len(dataloader）*args. batchsize 这个要比样本数多

model = torch.load('efficientNet-b0.pth')

ModuleNotFoundError: No module named 'model2'

不同img_cropped区域得到的关键点效果差距较大

非常感谢您开源的代码，但是当我使用不同大小的face crop时，得到的关键点效果差距比较大，可以请教下这是为什么吗？(下图紫色框是表示不同的face crop)

model.load_state_dict(torch.load('efficientNet-b0.pth'))

ModuleNotFoundError: No module named 'model2'

关于数据集处理

请问300w数据集是将 afw、helen、ibug、lfpw整个文件夹放在raw目录下吗，准备数据时总是会读不到图片?期待您的回复