ribosome-rbx/hood

Code for CVPR2023 paper "HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics"

HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics

Project   Paper  

This is a repository with training and inference code for the paper "HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics" (CVPR2023).

Latest update: 06.07.2023, code for computing metrics over the validation sequences added

Installation

Install conda enviroment

We provide a conda environment file hood.yml to install all the dependencies. You can create and activate the environment with the following commands:

conda env create -f hood.yml
conda activate hood

If you want to build the environment from scratch, here are the necessary commands:

Build enviroment from scratch

# Create and activate a new environment
conda create -n hood python=3.9 -y
conda activate hood

# install pytorch (see https://pytorch.org/)
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia -y

# install pytorch_geometric (see https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html)
conda install pyg -c pyg -y

# install pytorch3d (see https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md)
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install -c bottler nvidiacub -y
conda install pytorch3d -c pytorch3d -y


# install auxiliary packages with conda
conda install -c conda-forge munch pandas tqdm omegaconf matplotlib einops ffmpeg -y

# install more auxiliary packages with pip
pip install smplx aitviewer chumpy huepy

# create a new kernel for jupyter notebook
conda install ipykernel -y; python -m ipykernel install --user --name hood --display-name "hood"

Download data

HOOD data

Download the auxiliary data for HOOD using this link. Unpack it anywhere you want and set the HOOD_DATA environmental variable to the path of the unpacked folder. Also, set the HOOD_PROJECT environmental variable to the path you cloned this repository to:

export HOOD_DATA=/path/to/hood_data
export HOOD_PROJECT=/path/to/this/repository

SMPL models

Download the SMPL models using this link. Unpack them into the $HOOD_DATA/aux_data/smpl folder.

In the end your $HOOD_DATA folder should look like this:

$HOOD_DATA
    |-- aux_data
        |-- datasplits // directory with csv data splits used for training the model
        |-- smpl // directory with smpl models
            |-- SMPL_NEUTRAL.pkl
            |-- SMPL_FEMALE.pkl
            |-- SMPL_MALE.pkl
        |-- garment_meshes // folder with .obj meshes for garments used in HOOD
        |-- garments_dict.pkl // dictionary with garmentmeshes and their auxilliary data used for training and inference
        |-- smpl_aux.pkl // dictionary with indices of SMPL vertices that correspond to hands, used to disable hands during inference to avoid body self-intersections
    |-- trained_models // directory with trained HOOD models
        |-- cvpr_submission.pth // model used in the CVPR paper
        |-- postcvpr.pth // model trained with refactored code with several bug fixes after the CVPR submission
        |-- fine15.pth // baseline model without denoted as "Fine15" in the paper (15 message-passing steps, no long-range edges)
        |-- fine48.pth // baseline model without denoted as "Fine48" in the paper (48 message-passing steps, no long-range edges)

Inference

The jupyter notebook Inference.ipynb contains an example of how to run inference of a trained HOOD model given a garment and a pose sequence.

It also has examples of such use-cases as adding a new garment from an .obj file and converting sequences from AMASS and VTO datasets to the format used in HOOD.

Training

To train a new HOOD model from scratch, you need to first download the VTO dataset and convert it to our format.

You can find the instructions on how to do that and the commands used to start the training in the Training.ipynb notebook.

Validation Sequences

You can download the sequences used for validation (Table 1 in the main paper and Tables 1 and 2 in the Supplementary) using this link

You can find instructions on how to compute metrics over these sequences in the ComputeMetrics.ipynb notebook.

The code to generate these sequences yourself will be added soon.

Repository structure

See the RepoIntro.md for more details on the repository structure.

Citation

If you use this repository in your paper, please cite:

      @inproceedings{grigorev2022hood,
      author = {Grigorev, Artur and Thomaszewski, Bernhard and Black, Michael J and Hilliges, Otmar}, 
      title = {{HOOD}: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics}, 
      journal = {Computer Vision and Pattern Recognition (CVPR)},
      year = {2023},
      }