facebookresearch/omnimatterf

A matting method that combines dynamic 2D foreground layers and a 3D background model.

omnimatterf's Issues

About pre-trained models

Great project, can you provide pre-trained models please?

High resolution output with affordable memory

Great work!
I have two questions:

1. Resolution: if I want to render scenes with higher resolution (e.g. the solo video with 1080p), what should i do?
2. Memory: if i render high resolution, the cost of memory is too larger, are there some approach to save memory? (e.g. reduce cache feature?)

Very thanks!

errors in run_rodynrf

Thanks for your great work, I wonder where can I get
~/data/pretrained/midas/midas_v21-f6b98070.pt and ~/data/pretrained/raft/raft-things.pth are correct.
Thanks!

errors in run_depth.py

I tried downloaded dpt_beit_large_512.pt in https://github.com/isl-org/MiDaS, and it works, but it failed in OmnimatterRF's commands with following error messages:

    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DPTDepthModel:
        Unexpected key(s) in state_dict: "pretrained.model.blocks.0.attn.relative_position_index", "pretrained.model.blocks.1.attn.relative_position_index", "pretrained.model.blocks.2.attn.relative_position_index", "pretrained.model.blocks.3.attn.relative_position_index", "pretrained.model.blocks.4.attn.relative_position_index", "pretrained.model.blocks.5.attn.relative_position_index", "pretrained.model.blocks.6.attn.relative_position_index", "pretrained.model.blocks.7.attn.relative_position_index", "pretrained.model.blocks.8.attn.relative_position_index", "pretrained.model.blocks.9.attn.relative_position_index", "pretrained.model.blocks.10.attn.relative_position_index", "pretrained.model.blocks.11.attn.relative_position_index", "pretrained.model.blocks.12.attn.relative_position_index", "pretrained.model.blocks.13.attn.relative_position_index", "pretrained.model.blocks.14.attn.relative_position_index", "pretrained.model.blocks.15.attn.relative_position_index", "pretrained.model.blocks.16.attn.relative_position_index", "pretrained.model.blocks.17.attn.relative_position_index", "pretrained.model.blocks.18.attn.relative_position_index", "pretrained.model.blocks.19.attn.relative_position_index", "pretrained.model.blocks.20.attn.relative_position_index", "pretrained.model.blocks.21.attn.relative_position_index", "pretrained.model.blocks.22.attn.relative_position_index", "pretrained.model.blocks.23.attn.relative_position_index". 

Thank you very much for your support in advance!

For depth supervision command

Use the '+' Sign to Add New Configurations: If scene_scale is a new field that you are adding to the configuration, you should prefix it with a '+' sign in your command. This tells Hydra that you are adding a new field rather than modifying an existing one. Modify your command as follows:

python ./ui/cli.py train_ours wild/visd_3_rp_2_depth --use_depths -- fg_losses=[alpha_reg,brightness_reg,flow_recons,mask,recons,warped_alpha,bg_tv_reg,robust_depth_matching,bg_distortion] fg_losses.robust_depth_matching.config.alpha=0.1 fg_losses.bg_distortion.config.alpha=0.01 +data_sources.llff_camera.scene_scale=0.2

Can this project run on Windows systems?

How is Docker’s GPU passthrough supported under Windows systems?

environment:

• NVIDIA_DRIVER_CAPABILITIES=all

Preprocess Data

Great project, thank you to the author for their excellent work.
I encountered two issues during data preprocessing. One is the obtain COLMAP stage, and the other is the Depth Estimation stage. In the obtain COLMAP stage, only 10 of my images were pointed out; In the Depth Estimation stage, there is a problem of network structure mismatch in the weight dpt_beit_large_512.pt.

The first stage:

[2023-10-03 19:16:45,343][main][INFO] - Run: colmap feature_extractor --database_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/run_colmap/database.db --image_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/rgb_1x --SiftExtraction.use_gpu 0 --SiftExtraction.upright 0 --ImageReader.camera_model OPENCV --ImageReader.single_camera 1
[2023-10-03 19:16:45,418][main][INFO] - Run: colmap exhaustive_matcher --database_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/run_colmap/database.db --SiftMatching.use_gpu 0
[2023-10-03 19:16:45,543][main][INFO] - Run: colmap mapper --database_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/run_colmap/database.db --image_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/rgb_1x --output_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/run_colmap/sparse --Mapper.ba_refine_principal_point 1 --Mapper.filter_max_reproj_error 2 --Mapper.tri_complete_max_reproj_error 2 --Mapper.min_num_matches 32
Post-colmap
Cameras 5
hwf = 450 450 1505.2784385924213
Images # 10
Points (313, 3) Visibility (313, 10)
Depth stats 0.013273805525082845 11.832586834420141 10.408927046544216
Done with imgs2poses
[2023-10-03 19:28:29,752][main][ERROR] - Colmap only recovered 10 for 200 images

The last stage:

[2023-10-03 18:44:15,308][main][INFO] - Process 200 files
/data_1/anaconda3/envs/omnimatte/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Error executing job with overrides: ['input=/data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/rgb_1x', 'output=/data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/depth', 'model=/data_1/ldw_models/OmnimatteRF/data/pretrained/midas/dpt_beit_large_512.pt']
Traceback (most recent call last):
File "/data_1/ldw_models/OmnimatteRF/preprocess/run_depth.py", line 51, in main
model, transform, net_w, net_h = load_model(device, cfg.model, cfg.type, optimize=False)
File "/data_1/ldw_models/OmnimatteRF/third_party/MiDaS/midas/model_loader.py", line 52, in load_model
model = DPTDepthModel(
File "/data_1/ldw_models/OmnimatteRF/third_party/MiDaS/midas/dpt_depth.py", line 165, in init
self.load(path)
File "/data_1/ldw_models/OmnimatteRF/third_party/MiDaS/midas/base_model.py", line 18, in load
self.load_state_dict(parameters)
File "/data_1/anaconda3/envs/omnimatte/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1671, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DPTDepthModel:
Unexpected key(s) in state_dict: "pretrained.model.blocks.0.attn.relative_position_index", "pretrained.model.blocks.1.attn.relative_position_index", "pretrained.model.blocks.2.attn.relative_position_index", "pretrained.model.blocks.3.attn.relative_position_index", "pretrained.model.blocks.4.attn.relative_position_index", "pretrained.model.blocks.5.attn.relative_position_index", "pretrained.model.blocks.6.attn.relative_position_index", "pretrained.model.blocks.7.attn.relative_position_index", "pretrained.model.blocks.8.attn.relative_position_index", "pretrained.model.blocks.9.attn.relative_position_index", "pretrained.model.blocks.10.attn.relative_position_index", "pretrained.model.blocks.11.attn.relative_position_index", "pretrained.model.blocks.12.attn.relative_position_index", "pretrained.model.blocks.13.attn.relative_position_index", "pretrained.model.blocks.14.attn.relative_position_index", "pretrained.model.blocks.15.attn.relative_position_index", "pretrained.model.blocks.16.attn.relative_position_index", "pretrained.model.blocks.17.attn.relative_position_index", "pretrained.model.blocks.18.attn.relative_position_index", "pretrained.model.blocks.19.attn.relative_position_index", "pretrained.model.blocks.20.attn.relative_position_index", "pretrained.model.blocks.21.attn.relative_position_index", "pretrained.model.blocks.22.attn.relative_position_index", "pretrained.model.blocks.23.attn.relative_position_index".
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Could you help me? Thanks a lot.