g-u-n/motion-i2v

[SIGGRAPH 2024] Motion I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling

Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling

by Xiaoyu Shi^1*, Zhaoyang Huang^1*, Fu-Yun Wang^1*, Weikang Bian^1*, Dasong Li ¹, Yi Zhang¹, Manyuan Zhang¹, Ka Chun Cheung², Simon See², Hongwei Qin³, Jifeng Dai⁴, Hongsheng Li¹

¹CUHK-MMLab ²NVIDIA ³SenseTime ⁴ Tsinghua University

@article{shi2024motion,
            title={Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling},
            author={Shi, Xiaoyu and Huang, Zhaoyang and Wang, Fu-Yun and Bian, Weikang and Li, Dasong and Zhang, Yi and Zhang, Manyuan and Cheung, Ka Chun and See, Simon and Qin, Hongwei and others},
            journal={SIGGRAPH 2024},
            year={2024}
            }
}

motion-i2v's Issues

where is the inference code?

Hi, thank you for your amazing work on this project. However, I am having trouble locating the inference code within the repository. Could you please point me to the specific files or directories where the inference code is implemented?

Instructions on the project

Hi, thanks for sharing the great work. Could you provide more instructions on the project? (i.e. how to do inference, how to download checkpoints)

Is there any plan to release the checkpoint or the pretrain checkpoint

pretrained checkpoint :

output_dir: "outputs_ctrl_flow_gen"
pretrained_model_path: "models/stage1/StableDiffusion-FlowGen/"
vae_pretrained_path: "models/stage1/StableDiffusion-FlowGen/vae_flow/diffusion_pytorch_model.bin"
resumed_model_path: "models/stage1/StableDiffusion-FlowGen/unet/diffusion_pytorch_model.bin"