rq-wu/lamp

Official implement code of LAMP: Learn a Motion Pattern by Few-Shot Tuning a Text-to-Image Diffusion Model (Few-shot-based text-to-video diffusion)

武睿祺(Ruiqi Wu)

I am a master student of TMCC, College of Computer Science, Nankai University, China, under the supervision of Prof. Ming-Ming Cheng & Dr. Chun-Le Guo. My research interests are computer vision and machine learning, focusing on AIGC and low-level vision.

Visitor count

Star History

lamp's Issues

plans for Google Drive?

Hi there - amazing work! Just wondering when you are planning to upload the models to google drive - excited to play with them

Is one-shot learning possible?

Great Work!

I have a question about the setting. Is LAMP suitable for only few-shot learning or it is also suitable for one-shot learning?
In other words, does LAMP require 8~16 videos always or is one video okay too?

Thank you in advance.

Regarding the paper

Hi, thank you for the interesting work. I have a question about the proposed method.

a 2D convolution with an output channel of 1 along with a Sigmoid function is added

self.conv_gate = nn.Conv2d(out_channels, 1, 3, stride=1, padding=1)
x_gate = rearrange(x_2d, "b c f h w -> (b f) c h w")
c = x_gate.shape[1]
x_gate = self.sigmoid(self.conv_gate(x_gate)).repeat(1, c, 1, 1)

I would like to know what is the insight behind using a c -> 1 channel convolution and then repeating back c times. As a side question, what is the purpose of using a sigmoid function after this branch before multiplying to the conv_1d output?
Thanks.

About multi-action

Thank you for your excellent work! I try to train three actions at the same time(horse run, birds fly, waterfall), but the result is not as good as the single action. Can you give me some suggestions?

Evaluation code

Hi! I was wondering if you could share the evaluation code used in LAMP or point me to references that you used for the results reported in the paper? Thank you!

does each model only contain an individual motion or can multiple types of motion be trained inside a single model?

does each model only contain an individual motion or can you train multiple types of motion inside a single model like motiondirector ?

inference_script

Hi, thanks a lot for your interesting work! I know that in your paper you explain that you use the T2I model to generate the first frame during inference, but there doesn't seem to be any code in the "inference_script" that generates the first frame. I'm wondering if I'm mistaken.

How to inference for video editing?

Hi, Thanks for sharing the great work!
I woul dlike to learn how to try video editing after training?

can this do image to video

The script of evaluating the model

Thank you for your excellent work. I have noticed that you provided some functions to evaluate the model quantitatively. Can you provide a script to directly evaluate these metrics (i.e., alignment, consistency, diversity)? Thank you very much.

Question about the training time

Thanks for your great work! I have a question about the training time: when I train the horse_running in my GPU (RTX 3090 24G), it displays about 14hours for training. I want to know wheather this is normal?

Expecting your reply!

multi-gpu training

Thanks for your nice work! I want to ask that how to conduct multi-gpu training in your code. I set CUDA_VISIBLE_DEVICES=0,1, but it doesnot work. Hoping for your reply!

"Turn to smile" training data pwd is wrong