hongwenzhang/jvcr-3dlandmark

[TIP 2019] Adversarial Learning Semantic Volume for 2D/3D Face Shape Regression in the Wild

✨ Research Highlights ✨

🚩 Human/Hand/Face/Full-body Motion Capture:

👉 DaNet [TPAMI 2020]:

• Learning 3D Human Shape and Pose from Dense Body Parts
  ✨ DensePose to SMPL; part-based regressor; more robust to occlusion

👉 PyMAF [ICCV 2021, Oral] & PyMAF-X [TPAMI 2023]:

• PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
  ✨ better mesh-to-image alignment

• PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images
  ✨ an eXpressive version of PyMAF, supporting SMPL-X

👉 JVCR [TIP 2019]:

• Adversarial Learning Semantic Volume for 2D/3D Face Shape Regression in the Wild
  ✨ Joint voxel and coordinate regression

👉 ECT [TIFS 2018]:

• Combining Data-driven and Model-driven Methods for Robust Facial Landmark Detection
  ✨ Robust to occlusions and expressions

👉 More projects related to:

• clothed human reconstruction
• animatable avatar
• behavior understanding

jvcr-3dlandmark's Issues

progressbar error

Dear Sir:

which version of progressbar du you use？
there is some issue with progressbar 2.5 when running the train.py.

'''
bar = Bar('Processing', max=len(data_loader)) TypeError: init() got an unexpected keyword argument 'max'
'''
thanks

Can't unzip the downloaded pre-trained model

Hi Hongwen,

Thanks for share, but I can't unzip the model downloaded from google disk.
Any idea?

Training Failed

I created a virtual environment with python2.7 and the other required packages to run this. I was able to run the demo without issue, but when I go to run the training I get a conga line of errors! This is what I get in my terminal when I attempt to run python train.py --gpus 0 -j 4...

==> creating model: stacks=4, blocks=1, z-res=[1, 2, 4, 64]
coarse to fine mode: True
p2v params: 13.01M
v2c params: 19.46M
using ADAM optimizer.

Epoch: 1 | LR: 0.00025000
pre_training...
Traceback (most recent call last):
File "train.py", line 278, in
main(parser.parse_args())
File "train.py", line 90, in main
run(model, train_loader, mode, criterion_vox, criterion_coord, optimizer_G, optimizer_P)
File "train.py", line 144, in run
for i, (inputs, target, meta) in enumerate(data_loader):
File "/home/spacedorito/mypyenv2.7/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 345, in next
data = self._next_data()
File "/home/spacedorito/mypyenv2.7/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
return self._process_data(data)
File "/home/spacedorito/mypyenv2.7/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
data.reraise()
File "/home/spacedorito/mypyenv2.7/lib/python2.7/site-packages/torch/_utils.py", line 394, in reraise
raise self.exc_type(msg)
IOError: Caught IOError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/spacedorito/mypyenv2.7/lib/python2.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/spacedorito/mypyenv2.7/lib/python2.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/spacedorito/venv-for-landmark-detection/landmark_detection_training/datasets/fa68pt3D.py", line 100, in getitem
img = load_image(img_path) # CxHxW
File "/home/spacedorito/venv-for-landmark-detection/landmark_detection_training/utils/imutils.py", line 24, in load_image
return im_to_torch(scipy.misc.imread(img_path, mode='RGB'))
File "/home/spacedorito/mypyenv2.7/lib/python2.7/site-packages/numpy/lib/utils.py", line 101, in newfunc
return func(*args, **kwds)
File "/home/spacedorito/mypyenv2.7/lib/python2.7/site-packages/scipy/misc/pilutil.py", line 164, in imread
im = Image.open(name)
File "/home/spacedorito/mypyenv2.7/lib/python2.7/site-packages/PIL/Image.py", line 2766, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: 'data/300wLP/images/LFPW/LFPW_image_train_0626_13.jpg'

I've checked and confirmed that LFPW_image_train_0626_13.jpg is in the correct place. Any help is much appreciated.

License file

could you please add the license file to this repo.

A random bug

Hi everyone,

When I train the net, I got a random bug. An error will occur in random bench

---

Processing |########################## | (50860/61225) Data: 2.597300s | Batch: 3.278s | Total: 0:56:45 |Processing |########################## | (50880/61225) Data: 0.000299s | Batch: 0.681s | Total: 0:56:46 |Processing |########################## | (50900/61225) Data: 0.000489s | Batch: 0.691s | Total: 0:56:47 |Processing |########################## | (50920/61225) Data: 0.000502s | Batch: 0.683s | Total: 0:56:47 |Processing |########################## | (50940/61225) Data: 2.483688s | Batch: 3.165s | Total: 0:56:50 | ETA: 0:10:09 | LOSS vox: 0.0337; coord: 0.0034 | NME: 0.3116Traceback (most recent call last):
File "train.py", line 281, in
main(parser.parse_args())
File "train.py", line 90, in main
run(model, train_loader, mode, criterion_vox, criterion_coord, optimizer_G, optimizer_P)
File "train.py", line 144, in run
for i, (inputs, target, meta) in enumerate(data_loader):
File "/home/jliu9/anaconda3/envs/jvcr/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 623, in next
return self._process_next_batch(batch)
File "/home/jliu9/anaconda3/envs/jvcr/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 658, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/jliu9/anaconda3/envs/jvcr/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/jliu9/Codes/JVCR-3Dlandmark/datasets/fa68pt3D.py", line 151, in getitem
target_j = draw_labelvolume(target_j, tpts[j] - 1, self.sigma, type=self.label_type)
File "/home/jliu9/Codes/JVCR-3Dlandmark/utils/imutils.py", line 123, in draw_labelvolume
img[img_y[0]:img_y[1], img_x[0]:img_x[1]] = g[g_y[0]:g_y[1], g_x[0]:g_x[1]]
ValueError: could not broadcast input array from shape (7,7) into shape (7,8)

---

So, what's the problem?

Are there training code?

Hi, Zhang
Thank you for your great work.
Will you release the training code?

Thank you~

draw_labelvolume error

Dear Sir:
which version pytorch do you use?

when I'm training, got the below error.
'''
"work/JVCR-3Dlandmark/utils/imutils.py", line 119, in draw_labelvolume
img[img_y[0]:img_y[1], img_x[0]:img_x[1]] = g[g_y[0]:g_y[1], g_x[0]:g_x[1]]
ValueError: could not broadcast input array from shape (7,7) into shape (8,7)
'''

images in 300W_LP dataset donot have 3d landmarks

There are 2d landmarks in the label, but I don't find 3d landmark label.
Do I miss something?

.dat file

Yet I am working on project where I am using 68 face landmarks in my project. But its not 3d and more stable. So I am trying to make it more stable or finding new dataset. Can you tell me how can I make ".dat" file from this 3d dataset. Or can you share me .dat file for this dataset.

The issue for generating 3D heatmaps

Hi, Zhang

Thanks for your great work.

I try to generate ground truth volumes representation about 300W-LP based on your code utils.transforms.creat_volume and utils.imutils.draw_labelvolume.

But it does not same as your volumes representation generated from the net model you have trained,
could you tell me how to generate correct ground truth volumes representation for 300W-LP.

Thanks!

trying to implement the training parts

Hi Hongwen,

I'm trying to implement the traning parts about your algoirthm, but I can't get the correct results, the training didn't convergence. Can you send me a copy of your training code, I just use it for academic purpose.

Thanks zhaonan