justusschock/shapenet

PyTorch implementation of "Super-Realtime Facial Landmark Detection and Shape Fitting by Deep Regression of Shape Model Parameters" predicting facial landmarks with up to 400 FPS

shapenet

This repository contains the PyTorch implementation of our Paper "SUPER-REALTIME FACIAL LANDMARK DETECTION AND SHAPE FITTING BY DEEP REGRESSION OF SHAPE MODEL PARAMETERS".

Contents

• Installation
• Demo
• Usage
  • By Scripts
  • From Python
  • Pretrained Weights
• Our Paper

Installation

From Binary:

pip install shapenet

From Source:

pip install git+https://github.com/justusschock/shapenet

Demo

Demonstration Videos comparing our method to dlib can be found here as overlay and here as side-by-side view

Usage

By Scripts

For simplicity we provide several scripts to preprocess the data, train networks, predict from networks and export the network via torch.jit. To get a list of the necessary and accepted arguments, run the script with the -h flag.

Data Preprocessing

• prepare_all_data: prepares multiple datasets (you can select the datasets to preprocess via arguments passed to this script)
• prepare_cat_dset: Download and preprocesses the Cat-Dataset
• prepare_helen_dset: Preprocesses an already downloaded ZIP file of the HELEN Dataset (Download is recommended from here since this already contains the landmarks)
• prepare_lfpw_dset: Preprocesses an already downloaded ZIP file of the LFPW Dataset (Download is recommended from here since this already contains the landmarks)

Training

• train_shapenet: Trains the shapenet with the configuration specified in an extra configuration file (exemplaric configuration for all available datasets are provided in the example_configs folder)

Prediction

• predict_from_net: Predicts all images in a given directory (assumes existing groundtruths for cropping, otherwise the cropping to groundtruth could be replaced by a detector)

JIT-Export

• export_to_jit: Traces the given model and saves it as jit-ScriptModule, which can be accessed via Python and C++

From Python

This implementation uses the delira-Framework for training and validation handling. It supports mixed precision training and inference via NVIDIA/APEX (must be installed separately). The data-handling is outsourced to shapedata.

The following gives a short overview about the packages and classes.

shapenet.networks

The networks subpackage contains the actual implementation of the shapenet with bindings to integrate the ShapeLayer and other feature extractors (currently the ones registered in torchvision.models).

shapenet.layer

The layer subpackage contains the Python and C++ Implementations of the ShapeLayer and the Affine Transformations. It is supposed to use these Layers as layers in shapenet.networks

shapenet.jit

The jit subpackage is a less flexible reimplementation of the subpackages shapenet.networks and shapenet.layer to export trained weights as jit-ScriptModule

shapenet.utils

The utils subpackage contains everything that did not suit into the scope of any other package. Currently it is mainly responsible for parsing of configuration files.

shapenet.scripts

The scripts subpackage contains all scipts described in Scripts and their helper functions.

Pretrained Weights

Currently Pretrained Weights are available for grayscale faces and cats.

For these Networks the image size is fixed to 224 and the pretrained weights can be loaded via torch.jit.load("PATH/TO/NETWORK/FILE.ptj"). The inputs have to be of type torch.Tensor with dtype torch.float in shape (BATCH_SIZE, 1, 224, 224) and normalized in a range between (0, 1).

Our Paper

If you use our Code for your own research, please cite our paper:

@article{Kopaczka2019,
title = {Super-Realtime Facial Landmark Detection and Shape Fitting by Deep Regression of Shape Model Parameters},
author = {Marcin Kopaczka and Justus Schock and Dorit Merhof},
year = {2019},
journal = {arXiV preprint}
}

The Paper is available as PDF on arXiv.

shapenet's Issues

question about predict

Hi:
I use the default way to train the net .When I predict, I only replace the lmk_bounds with a face detector,but the landmarks not fit good.Is there anything I did wrong?
And for last question about train,I find the shape data cannot set backend, so I set__BACKENDS = ['TORCH'] and it worked. Thanks for you help!

Config reading error: 'str' object has no attribute 'items'

Hi.
when I am trying to train the network, I have got the following error.
Do you have any tips?

conda
python 3.6
PyYAML 5.1

AttributeError: 'str' object has no attribute 'items'
for doc in docs: Traceback (most recent call last): File "./scripts/train_single_shapenet.py", line 143, in <module> train_shapenet() File "./scripts/train_single_shapenet.py", line 38, in train_shapenet config_dict = config(os.path.abspath(args.config)) File "C:\Users\User\Anaconda3\envs\Env_python3_shapenet\lib\site-packages\shapenet\utils\load_config_file.py", line 54, in __call__ for key, vals in group_dict.items(): AttributeError: 'str' object has no attribute 'items'

error for prediction: reading weights

**hi

My model is trained but when I am using the predict_from_net.py with the wieghts,
checkpoint_best.pth,
I am getting following errors. Is there any suggestions?
Thanks in advance.**

for doc in docs:
Traceback (most recent call last):
  File "./scripts/predict_from_net.py", line 35, in predict
    net = torch.jit.load(os.path.abspath(args.weight_file))
  File "C:\Users\User\Anaconda3\envs\Env_python36_shapenet\lib\site-packages\torch\jit\__init__.py", line 146, in load
    torch._C.import_ir_module(module_lookup, f, map_location, _extra_files)
RuntimeError: [enforce fail at ..\caffe2\serialize\inline_container.cc:137] . PytorchStreamReader failed reading zip archive: failed
finding central directory
(no backtrace available)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./scripts/predict_from_net.py", line 60, in predict
    os.path.abspath(args.weight_file))["state_dict"]["model"])
KeyError: 'state_dict'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./scripts/predict_from_net.py", line 139, in <module>
    predict()
  File "./scripts/predict_from_net.py", line 62, in predict
    net.load_state_dict(torch.load(os.path.abspath(args.weight_file)))
  File "C:\Users\User\Anaconda3\envs\Env_python36_shapenet\lib\site-packages\torch\nn\modules\module.py", line 777, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for ShapeNetwork:
        Missing key(s) in state_dict: "_out_layer._indices_shape_params", "_out_layer._indices_homogen_params", "_out_layer._shape_layer._layer._shape_mean", "_out_layer._shape_layer._layer._shape_components", "_out_layer._homogen_trafo._indices_scale_params", "_out_layer._homogen_trafo._indices_rotation_params", "_out_layer._homogen_trafo._indices_translation_params", "_out_layer._homogen_trafo._layer._trafo_matrix", "_model.model.conv_1._conv.weight", "_model.model.conv_1._conv.bias", "_model.model.conv_2._conv.weight", "_model.model.conv_2._conv.bias", "_model.model.down_conv_1._conv.weight", "_model.model.down_conv_1._conv.bias", "_model.model.conv_3._conv.weight", "_model.model.conv_3._conv.bias", "_model.model.conv_4._conv.weight", "_model.model.conv_4._conv.bias", "_model.model.down_conv_2._conv.weight", "_model.model.down_conv_2._conv.bias", "_model.model.conv_5._conv.weight", "_model.model.conv_5._conv.bias", "_model.model.conv_6._conv.weight", "_model.model.conv_6._conv.bias", "_model.model.down_conv_3._conv.weight", "_model.model.down_conv_3._conv.bias", "_model.model.conv_7._conv.weight", "_model.model.conv_7._conv.bias", "_model.model.conv_8._conv.weight", "_model.model.conv_8._conv.bias", "_model.model.down_conv_4._conv.weight", "_model.model.down_conv_4._conv.bias", "_model.model.conv_9._conv.weight", "_model.model.conv_9._conv.bias", "_model.model.conv_10._conv.weight", "_model.model.conv_10._conv.bias", "_model.model.conv_11._conv.weight", "_model.model.conv_11._conv.bias", "_model.model.conv_12._conv.weight", "_model.model.conv_12._conv.bias", "_model.model.final_conv.weight", "_model.model.final_conv.bias".
        Unexpected key(s) in state_dict: "optimizer", "model", "epoch".

wrong angle

Hi I got some predictions which have wrong rotation angle.
Is there suggestions how I can work on that?

thank you

A trainging problem

After run the train_single_shapenet.py ,it returns a attribute error like this:
Traceback (most recent call last):
File "scripts/train_single_shapenet.py", line 144, in
train_shapenet()
File "scripts/train_single_shapenet.py", line 19, in train_shapenet
from delira.training import PyTorchNetworkTrainer
File "/anaconda3/lib/python3.6/site-packages/delira/training/init.py", line 2, in
from .experiment import AbstractExperiment
File "/anaconda3/lib/python3.6/site-packages/delira/training/experiment.py", line 7, in
from ..models import AbstractNetwork
File "/anaconda3/lib/python3.6/site-packages/delira/models/init.py", line 7, in
from .classification import VGG3DClassificationNetworkPyTorch,
File "/anaconda3/lib/python3.6/site-packages/delira/models/classification/init.py", line 8, in
from .classification_network_tf import ClassificationNetworkBaseTf
File "/anaconda3/lib/python3.6/site-packages/delira/models/classification/classification_network_tf.py", line 5, in
from delira.models.classification.ResNet18 import ResNet18
File "/anaconda3/lib/python3.6/site-packages/delira/models/classification/ResNet18.py", line 6, in
relu = tf.keras.layers.ReLU
AttributeError: module 'tensorflow.keras.layers' has no attribute 'ReLU'

How can I solve this problem?

about the PCA on the training set

Hi,I read the paper on shapenet, but I have a problem, that is, when you perform PCA of landmarks on the training set, the shape of all samples is used as a whole set to perform PCA and then take the eigenvector. Is it? If so, how do you find the weight of each eigenvector for each sample?

multiprocessing.pool error

Hi.
When I am using the prepare_datasets I got following error:
windows 10,
conda,
pip shapenet,

Is there any hints for fixing this?
Thanks in advance.

File "./scripts/prepare_datasets.py", line 397, in prepare_all_data normalize_pca_rot=args.normalize_pca_rot, File "./scripts/prepare_datasets.py", line 202, in _prepare_ibug_dset normalize_rot=normalize_pca_rot, rotation_idxs=(37, 46)) File "./scripts/prepare_datasets.py", line 36, in _make_pca data = SingleShapeDataProcessing.from_dir(data_dir) File "C:\Users\User\Anaconda3\envs\Env_python3_shapenet\lib\site-packages\shapedata\single_shape\data_processing.py", line 383, in from_dir samples = p.map(SingleImage2D.from_files, files) File "C:\Users\User\Anaconda3\envs\Env_python3_shapenet\lib\multiprocessing\pool.py", line 266, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "C:\Users\User\Anaconda3\envs\Env_python3_shapenet\lib\multiprocessing\pool.py", line 644, in get raise self._value multiprocessing.pool.MaybeEncodingError: Error sending result: '[<shapedata.single_shape.data_processing.SingleImage2D object at 0x000001C17D4F0CC0>, <shapedata.single_shap ...

pca components not fully loaded

Assuming we have 50 keypoints, which are 100 numbers.

In 'prepare_datasets', pca components are computed by pca.fit() with shape(100100). the shape of pca.mean_ is 1100. pca.mean_ + pca.components = shape(101, 100), which are resized to (101, 50, 2).

But in 'train_single_shapenet', we only load first 51502, which is not well loaded. Why is that? Is this we only need top 50 components? If I load full components, can I get a better result?

shapes = np.load(os.path.abspath(config_dict["layer"].pop("pca_path"))
         )["shapes"][:config_dict["layer"].pop("num_shape_params") + 1]

installation error

I got this error during installation

Collecting torch>=1.0.0; extra == "torch" (from delira[torch]<=0.3.1->shapenet==0.1.0)
ERROR: Could not find a version that satisfies the requirement torch>=1.0.0; extra == "torch" (from delira[torch]<=0.3.1->shapenet==0.1.0) (from versions: 0.1
.2, 0.1.2.post1)
ERROR: No matching distribution found for torch>=1.0.0; extra == "torch" (from delira[torch]<=0.3.1->shapenet==0.1.0)

didn't find a solution yet
can someone help?

problem with prediction

Does the code always a need a "landmark.pts" files to detect the final face?
why can't i just detect the face using a direct input image?
Can you please clarify if i am doing something wrong here?

error in script predict_from_net.py

In ./shapenet/scripts/predict_from_net.py line 118
pred = net(img_tensor).cpu().numpy()["pred"][0]
should be
pred = net(img_tensor).cpu().numpy()[0]

obviously net(img_tensor).cpu().numpy() is not a dict

Embed in mobile as dlib

Hi,

Is there a way to run this lib on mobile?
Dlib runs on Android and iOS but it's written in C++.
Any idea on how to run it on mobile devices?

Thanks

Grayscale faces

I'm curious why the faces are converted to grayscale during training and prediction.

Is there a disadvantage to using faces in the three-channel RGB format?

question: how to get values of training losses

Hi
I am not very familiar with delira pytorch. Could you kindly let us know, how we can plot or the get values of losses (train/val) of training procedure? is it in embedded in checkpoint_best.pth?
Thanks in advance.

delira.training.PyTorchNetworkTrainer missing argument

Hi,
I'm getting an error while running train_single_shapenet.py. It says that a required positional argument (key_mapping) is missing in the function call. Please find the output below:

$ python train_single_shapenet.py -c ~/shapenet/shapenet/example_configs/helen.config -v

WARNING: Logging before flag parsing goes to stderr.
W0726 12:15:12.281908 139805229729536 deprecation_wrapper.py:119] From /home/anand/env_16.04_py3/lib/python3.5/site-packages/delira/models/abstract_network.py:251: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

/home/anand/env_16.04_py3/lib/python3.5/site-packages/shapenet/utils/load_config_file.py:50: YAMLLoadWarning: calling yaml.load_all() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
for doc in docs:
Number of Parameters: 6248029
Traceback (most recent call last):
File "train_single_shapenet.py", line 141, in
train_shapenet()
File "train_single_shapenet.py", line 71, in train_shapenet
**config_dict["training"])
TypeError: init() missing 1 required positional argument: 'key_mapping'

error while training

Hello, during training in validation step I get this error. Any ideas what is happening?
Epoch 1: 100%|████████████████████████████| 689/689 [03:18<00:00, 3.75 batch/s]
Validate: 0%| | 0/689 [00:00<?, ? sample/s]Traceback (most recent call last):
File "shapenet/scripts/train_single_shapenet.py", line 149, in
train_shapenet()
File "shapenet/scripts/train_single_shapenet.py", line 145, in train_shapenet
val_score_mode='lowest')
File "/home/vilius/anaconda3/envs/pytorch/lib/python3.7/site-packages/delira/training/base_trainer.py", line 444, in train
verbose=verbose))
File "/home/vilius/anaconda3/envs/pytorch/lib/python3.7/site-packages/delira/training/predictor.py", line 291, in predict_data_mgr_cache_metrics_only
cache_preds=False, **kwargs)
File "/home/vilius/anaconda3/envs/pytorch/lib/python3.7/site-packages/delira/training/predictor.py", line 399, in predict_data_mgr_cache
**kwargs):
File "/home/vilius/anaconda3/envs/pytorch/lib/python3.7/site-packages/delira/training/predictor.py", line 219, in predict_data_mgr
preds = self.predict(copy.copy(batch_dict), **kwargs)
File "/home/vilius/anaconda3/envs/pytorch/lib/python3.7/site-packages/delira/training/predictor.py", line 138, in predict
**pred
TypeError: convert_torch_tensor_to_npy() argument after ** must be a mapping, not Tensor

Cat-landmark detection

Hi,
I'm trying to use cat-landmark model.
Here is my code:

import numpy as np
import torch
from PIL import Image
import cv2

model = torch.jit.load("pretrained_cats.ptj")
model.eval()
model.cpu()

img = Image.open("dataset/poszukiwarka_kotow_splitted/train/Kot_62_Karinka/b.jpg").convert('L')
img_resized = img.resize((224,224))
img_preprocessed = np.array(img_resized).astype(np.float32)/255.
img_preprocessed = img_preprocessed[np.newaxis, np.newaxis, :, :]
print(np.max(img_preprocessed), np.min(img_preprocessed))

img_tensor = torch.from_numpy(img_preprocessed)
landmarks = model(img_tensor)
landmarks = landmarks.detach().numpy()[0]

img_resized_land = np.array(img_resized)
for x,y in landmarks:
    cv2.circle(img_resized_land, (x,y), 2,(255), -1)

input image:

output:

Result look like random one. What am I doing wrong?

inference with no lmk_bounds

I have tried predict_from_net, which use a lmk_bounds to predict. If I don't have a lmk_bounds, how to predict on a new image?

lmk_bounds = sample.get_landmark_bounds(sample.lmk)
min_y, min_x, max_y, max_x = lmk_bounds
range_x = max_x - min_x
range_y = max_y - min_y

max_range = max(range_x, range_y) * (1 + crop)

center_x = min_x + range_x / 2
center_y = min_y + range_y / 2

tmp = sample.crop(center_y - max_range / 2,
                center_x - max_range / 2,
                center_y + max_range / 2,
                center_x + max_range / 2)

EDIT: Adjusted Code Formatting

Does not the PCA operation leak landmarks to the training process?

Sorry, I have some doubts, how do you make the PCA process in predict stage? I read your code but still can not understand. And what is the target in training stage where you apply PCA with landmark points?

Test, CI/CD, Coverage

• Add Tests
• Add Travis CI
• Add Coverage

Prediction time compared with dlib

Hi,

Previously, you had mentioned that your method is faster than dlib. However, when I downloaded the pretrained model and ran it on videos, I found that it was much slower than dlib. I used the same code (my own test app) and face detector (mobilenet SSD) to test shapenet and dlib.

question: About the number of PCA components

In your experimental environment, you have 68 landmarks for each face and you set the PCA components to 25 in the config file. Based on your experience, what would be a suitable choice of PCA components if we have just 23 landmarks in each image?
Thanks in advance.

Question about predict_from_net.py

This is my parameter：
args = easydict.EasyDict({
"visualize": True,
"in_path": './inputs',
"out_path": './outputs',
"weight_file": './pretrained_face.ptj',
"config_file": './example_configs/lfpw.config'
})

I encountered such a problem when I was running.
TypeError Traceback (most recent call last)
in
92 print(_data.lmk)
93 pred = process_sample(_data, img_size=config_dict["data"]["img_size"], net=net,
---> 94 device=device)
95
96 fname = os.path.split(_data.img_file)[-1].rsplit(".", 1)[0]

in process_sample(sample, img_size, net, device, crop)
45 def process_sample(sample, img_size, net, device, crop=0.1):
46 print(sample)
---> 47 lmk_bounds = sample.get_landmark_bounds(sample.lmk)
48 min_y, min_x, max_y, max_x = lmk_bounds
49 range_x = max_x - min_x

C:\ProgramData\Anaconda3\lib\site-packages\shapedata\base_data_processing.py in get_landmark_bounds(lmks)
815 """
816
--> 817 min_y = lmks[:, 0].min()
818 max_y = lmks[:, 0].max()
819 min_x = lmks[:, 1].min()

TypeError: 'NoneType' object is not subscriptable

it seems that
_data = SingleShapeSingleImage2D.from_files(file)
did not read and process the data

I want to apply the existing model directly. Is there any problem with my parameter settings?

Is there a video Demo to show it's performance?

Invalid syntax error

Hi . I run the code.
But there is a following error in shape_network.py:
def model(self, model: torch.nn.Module):
^
SyntaxError: invalid syntax

Landmarks accuracy on lmk_bounds

I have experimented with lmk_bounds in two scenarios :

1. Replaced lmk_bounds to the bounds of the face bounding box given by the face detector
2. Replaced lmk_bounds to the bounds of the image with only one face [Loosely cropped to the face - see the figure below]

In both the above mentioned scenarios the landmarks deviate from the ground truths drastically. Is this expected? Also can you please tell me your system configuration where you ran your code: I am able to achieve 3fps on my machine - GeForce GTX 960M 4GB with cuda-9

Training Question

Hello,
I found your answer in #31.
Pre-trained weight of Thermal faces are still not available?
I downloaded ThermalFace Database, and about to train.
Shall I duplicate config file of helen or some adjustment is required?

tf.train.Optimizer error

When I have tensorflow v2.0 in my environment and import shapenet, the tf.train.Optimizer doesn't exist error occurred. Seems like it's a compatibility issue. I downgrade my tensorflow to version 1.14.0, I can still see this warning The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

pretrained weights, link broken.

Hi, and thank you for making this code available.

the links for the dlib compare videos, and the pretrained weights, are broken.
would it be possible for you to re-link them?

Thanks!

can this be used for realtime detection of basic training

Iam looking for a ml code for realtime shape detection for my final year projects only need to detect basic shapes like triangle ,square,circle etc and control a robotic arm based on detection.can u please help me out iam new to programming or ml as iam an instrumentation student please help.

could not compile: invalid syntax(predict_from_net.py, line 54)

[BUG]AttributeError: img_size not available for jitted models

What is the PyTorch version you used for training?
Im not able to restore the weights, i get this error:
AttributeError: 'ScriptModule' object has no attribute 'img_size'