whaohan/pivotmesh

Official code for paper: PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidance

PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidance

Project Page | Paper | Weight

Preparation

Install the packages in requirements.txt. The code is tested under CUDA version 12.1.

conda create -n pivotmesh python=3.9
conda activate pivotmesh
pip install -r requirements.txt

Training

# train AE
accelerate launch --mixed_precision=fp16 train_AE.py 

# train PivotMesh
accelerate launch --mixed_precision=fp16 train_pivotmesh.py

Inference

python pivot_infer.py \
    --model_path checkpoints/PivotMesh-objaversexl/mesh-transformer.ckpt.ft.50.pt \
    --AE_path checkpoints/AE-objaversexl/mesh-autoencoder.ckpt.72.pt \
    --output_path output/PivotMesh \
    --dataset_name objaverse \
    --batch_size 16 \
    --sample_num 1 \
    --temperature 0.5 \
    --pivot_rate 0.1 \
    --condition no   # 'no' for unconditional, 'pivot' for conditional 

Evaluation

# evaluate the performance on PivotMesh
python evaluate.py

Acknowledgement

• MeshGPT
• HyperDiffusion

Citation

@misc{weng2024pivotmesh,
    title={PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidance}, 
    author={Haohan Weng and Yikai Wang and Tong Zhang and C. L. Philip Chen and Jun Zhu},
    year={2024},
    eprint={2405.16890},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

pivotmesh's Issues

RuntimeError: Error(s) in loading state_dict for MeshAutoencoder

Thank you for the great work. Could you please check the following error that occurred while loading the pre-trained weights?

    autoencoder = MeshAutoencoder.init_and_load(args.AE_path)
  File "/root/anaconda3/envs/pivotmesh/lib/python3.9/site-packages/pytorch_custom_utils/save_load.py", line 75, in _init_and_load_from
    _load(model, path, strict = strict)
  File "/root/anaconda3/envs/pivotmesh/lib/python3.9/site-packages/pytorch_custom_utils/save_load.py", line 58, in _load
    self.load_state_dict(pkg['model'], strict = strict)
  File "/root/anaconda3/envs/pivotmesh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 2189, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for MeshAutoencoder:
	Unexpected key(s) in state_dict: "encoder.layers.0.0.0.bias", "encoder.layers.1.0.0.bias", "encoder.layers.2.0.0.bias", "encoder.layers.3.0.0.bias", "encoder.layers.4.0.0.bias", "encoder.layers.5.0.0.bias", "encoder.layers.6.0.0.bias", "encoder.layers.7.0.0.bias", "encoder.layers.8.0.0.bias", "encoder.layers.9.0.0.bias", "encoder.layers.10.0.0.bias", "encoder.layers.11.0.0.bias", "encoder.layers.12.0.0.bias", "encoder.layers.13.0.0.bias", "encoder.layers.14.0.0.bias", "encoder.layers.15.0.0.bias", "encoder.layers.16.0.0.bias", "encoder.layers.17.0.0.bias", "encoder.layers.18.0.0.bias", "encoder.layers.19.0.0.bias", "encoder.layers.20.0.0.bias", "encoder.layers.21.0.0.bias", "encoder.layers.22.0.0.bias", "encoder.layers.23.0.0.bias", "encoder.final_norm.bias", "decoder_coarse.layers.0.0.0.bias", "decoder_coarse.layers.1.0.0.bias", "decoder_coarse.layers.2.0.0.bias", "decoder_coarse.layers.3.0.0.bias", "decoder_coarse.layers.4.0.0.bias", "decoder_coarse.layers.5.0.0.bias", "decoder_coarse.layers.6.0.0.bias", "decoder_coarse.layers.7.0.0.bias", "decoder_coarse.layers.8.0.0.bias", "decoder_coarse.layers.9.0.0.bias", "decoder_coarse.layers.10.0.0.bias", "decoder_coarse.layers.11.0.0.bias", "decoder_coarse.final_norm.bias", "decoder_fine.layers.0.0.0.bias", "decoder_fine.layers.1.0.0.bias", "decoder_fine.layers.2.0.0.bias", "decoder_fine.layers.3.0.0.bias", "decoder_fine.layers.4.0.0.bias", "decoder_fine.layers.5.0.0.bias", "decoder_fine.layers.6.0.0.bias", "decoder_fine.layers.7.0.0.bias", "decoder_fine.layers.8.0.0.bias", "decoder_fine.layers.9.0.0.bias", "decoder_fine.layers.10.0.0.bias", "decoder_fine.layers.11.0.0.bias", "decoder_fine.final_norm.bias".

Looking forward to the code

Hi, I'm very interested in this paper PivotMesh, and wonder when can it be opensourced.

Preparation of Training Data and Missing File Issue

Thank you for sharing the code; it has been very inspiring!

Could you please advise on how I should prepare the training data? Is there a corresponding script for this purpose?

FileNotFoundError: [Errno 2] No such file or directory: './data/objaverse-lp-500/train-500'

Generate imcomplete meshes

Hi, I'm using the released weight provided in the repository for inferencing new laptop meshes, but it generated with incomplete meshed.

The above image is a screenshot taken under blender, and the middle one is the ground truth I founded at https://sketchfab.com/3d-models/gaming-laptop-computer-391952ba56ee4a4f8effdaf78afc67cb. This mesh should be included by Objaverse-XL, so I'm using the inference command provide in the repository directly.

I am not sure why I got those poor results, any suggestions on that?

Thanks for your help in advance.