pansanity666/transhuman

Official code for ICCV 2023 paper: "TransHuman: A Transformer-based Human Representation for Generalizable Neural Human Rendering".

News

• 05/08/2023 The code and the pretrained model are released!
• 23/10/2023 We have collected recent advances in Human Avatars in this repo, welcome stars!

TransHuman

Official code for ICCV 2023 paper:

TransHuman: A Transformer-based Human Representation for Generalizable Neural Human Rendering
Xiao Pan^1,2, Zongxin Yang¹, Jianxin Ma², Chang Zhou², Yi Yang^1
1 ReLER Lab, CCAI, Zhejiang University; ² Alibaba DAMO Academy

[Project Page | arXiv]

• We present a brand-new framework named TransHuman for generalizable neural human rendering, which learns the painted SMPL under the canonical space and captures the global relationships between human parts with transformers.

• We achieve SOTA performance on various settings and datasets.

• We also have better efficiency.

Environment

We test with:

• python==3.6.12
• pytorch==1.10.2
• cuda==11.3

# under TransHuman dir

# create conda environment
conda create -n transhuman python=3.6 
conda activate transhuman

# make sure that the pytorch cuda is consistent with the system cuda
# install pytorch via conda
https://pytorch.org/get-started/locally/

# install requirements 
pip install -r requirements.txt

# install pytorch3d (we build from source)
https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md

Dataset Preparation

ZJU-MoCap

Please follow NHP to prepare the ZJU-MoCap dataset.

The final structure of data folder should be:

# under TransHuman dir
-data
  -smplx
    - smpl
    - ...
  -zju_mocap
    - CoreView_313
    - ...
  -zju_rasterization
    - CoreView_313
    - ...

TODO: Code for more datasets are coming.

Training

sh ./scripts/train.sh

The checkpoints will be saved under ./data/trained_model/transhuman/$EXP_NAME .

Evaluation

sh ./scripts/test.sh $GPU_NUMBER $EPOCH_NUMBER $EXP_NAME 

The config of different settings are provided in test.sh. Modify them as you need.

For reproducing the results in the paper, please download the official checkpoints from here.

Put it under ./data/trained_model/transhuman/official, and run:

sh ./scripts/test.sh 0 2100 official 

Visualization

Free-viewpoint video

1. Render the free-viewpoint frames via running:

   sh ./scripts/video.sh $GPU_NUMBER $EPOCH_NUMBER $EXP_NAME  

   The rendered frames will be saved under ./data/perform/$EXP_NAME.

2. Use gen_freeview_video.py for getting the final video.

Mech reconstruction

1. Extract the mesh via running:

   sh ./script/mesh.sh $GPU_NUMBER $EPOCH_NUMBER $EXP_NAME  

   The meshes will be saved under ./data/mesh/$EXP_NAME .

2. Render the meshes using render_mesh_dynamic.py. The rendered frames will also be saved under ./data/mesh/$EXP_NAME

3. Use gen_freeview_video.py for getting the final video.

Citation

If you find our work useful, please kindly cite:

@InProceedings{Pan_2023_ICCV,
    author    = {Pan, Xiao and Yang, Zongxin and Ma, Jianxin and Zhou, Chang and Yang, Yi},
    title     = {TransHuman: A Transformer-based Human Representation for Generalizable Neural Human Rendering},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {3544-3555}
}

Contact

For questions, feel free to contact [email protected].

Acknowledgments

This project is mainly based on the code from NHP and humannerf. We also thank Sida Peng of Zhejiang University for helpful discussions on details of ZJU-MoCap dataset.

transhuman's Issues

About training epoch

Hi! Thanks for your amazing work!
Recently, we retrained your code on thuman2 dataset.
For sanity checks, we train your code on ZJU-mocap, but we don't get meaningful images after 50 epochs. We just got some all-black images.

How many epochs do we need to get a meaningful output image during training? Could you provide more training details?
Thank you in advance.

Release date of the code

Hi, thanks for your contributions about human neural rendering. When would you like to release the code?

Question of Coordinate Encoding

Thanks for your great work!
Actually I am being confused with Eq.4 in the paper:

It seem that W_i^o was used to transform query point p from world coordinate to token coordinate in observation space.
However, W_i^o is obtained from the rotation matrix of smpl vertices, which is used to transform vertices from vertices coordinate to world coordinate in observation space.
Therefore, I think Eq.4 should be the following:

For this formula, could there be any errors in my understanding?

Some wonder about function "voxelization" in "if_clight_renderer.py"

Hi, thanks for your great work.

I'm trying to run your code on RTX3090 gpu, however I found it meets "CUDA OUT OF MEMORY" when executing "loss.backward". I think the reason may bethe accumulated grads, so I add the code "with torch.no_grad()" in function "voxelization", and it works.
Following is the changed code:

def voxelization(self, dict_voxel2pc_ind, SMPL, return_list=False):
        # voxelize based on dict

        voxel_list = []
        voxel = []
        with torch.no_grad(): #additional code
            for vn, pc_list in dict_voxel2pc_ind.items():
                # mean pooling 
                voxel.append(SMPL[pc_list].mean(0))
                voxel_list.append(SMPL[pc_list])
        voxel = torch.stack(voxel)  # torch.Size([2115, 67])

        if return_list:
            return voxel, voxel_list
        else:
            return voxel

So I wonder will it affect the final performance? Thanks for your reading.

About trainning time and GPU memory

"What is the total training time for this method, and what GPU model is required?

Bug in test

Hi! We encounter a bug when we test on THuman Dataset.
It only occurs in extremely special circumstances.
https://github.com/pansanity666/TransHuman/blob/759b220410370f5ad322ecc68104414dd6401e29/lib/networks/cross_transformer.py#L298C28-L298C28
When alpha shape is [1, 1, 1], squeeze() will cause error. (It happen when density_mask only have one valid value)
We add unsqueeze() to avoid this bug and it work well.

Single GPU training

Hi,I only have one RTX 3090 for training,how should I modify the script to turn of the DDP training?