Batch/Video Processing

[advimman]

Batch/Video Processing

[windj007]

LaMa was designed for image inpainting. Technologies for video inpainting significantly differ from those for separate images, because in videos on the one hand you need temporal consistency and on the other you have more information - regions of different frames can be reused.

LaMa can be used for videos, but the quality will probably be inferior compared to designated video inpainting methods. The most straightforward way to use existing LaMa codebase to process video is:

1. Split video to separate frames and store frames to a folder, with something like ffmpeg -i video.mp4 video_frames/%05d.jpg. Somehow add masks for each frame to the same folders.
2. Run prediction with LaMa - its current codebase can process all images in a folder in a single run.
3. Merge inpainted images into a video using e.g. ffmpeg.

Note that you need to specify a mask for each separate frame. There is no functionality for object segmentation and tracking in videos in LaMa.

There is a third-party colab with CLIP-based segmentation and LaMa for video inpainting, which is based on the similar idea (twitter post and video, colab)

There are designated video completion methods, e.g. Flow-edge guided video completion.

[Echolink50]

Thank you so much for a very clear and informative answer.