Upscaled RBG and CCM ,Tile-Based generation

[thu-ml]

Upscaled RBG and CCM ,Tile-Based generation

[thuwzy]

Thank you for your interest in our CRM paper!

1. The decoder cannot directly work at resolution 512x3,072. However, I will upload the model trained on this resolution 512x3,072 if you can get the upsampling image work.
2. The CCM is for better geometry and cannot be skipped. I have conducted ablation study on Figure 10 in my paper. CRM without CCM input has worse geometry.
3. Actually depth map can be equivalently transformed to CCM in my framework. So I think it is highly likely to work.
   By the way, I think the resolution of CCM is not very important. I think a good pipeline may be generate 256*1536 image and CCM, and then use neural network to upsample the image and simply resize the CCM to be in the resolution of 512x3,072.

[mr-lab]

original model render:

couple of Re-renders


your work is a blessing to us, those are Re-renders of the RGB to retexture the mesh . more consistency is needed .
will move to depth map after that , good depth comes from good RGB.
cheers.

[mr-lab]

thank you very much will be waiting for that model
I will explore more Point 3 .

[mosvlad]

original model render: couple of Re-renders your work is a blessing to us, those are Re-renders of the RGB to retexture the mesh . more consistency is needed . will move to depth map after that , good depth comes from good RGB. cheers.

Awesome!!! Can you share your result with code?

[zz7379]

original model render: couple of Re-renders your work is a blessing to us, those are Re-renders of the RGB to retexture the mesh . more consistency is needed . will move to depth map after that , good depth comes from good RGB. cheers.

is this a up-scaled rgb? or rendered mesh?

[mosvlad]

I'm trying to upscale RGB from stage1

This:

CRM/run.py

Line 152 in 3e677cb

stage1_images = rt_dict["stage1_images"]

and this:

CRM/pipelines.py

Line 135 in 3e677cb

stage1_images = self.stage1_sample(pixel_img, prompt, scale=scale, step=step)

For upscale i'm used BSRGAN.
https://github.com/cszn/BSRGAN

1. Generate images (256x1536) by stage 1
2. Upscale it by BSRGAN (x2 or 4x)
3. Resize images to (256x1536)
4. Use upscaled and resized images for generate3d

This steps not make quality improvement like @mr-lab comments.

Another way i tried make upscale for every image generated in step1:

1. Generate image (256x256)
2. Upscale it by BSRGAN (x2 or x4)
3. Resize to original size (256x256)
4. Use upscaled and resized images for stage2

The results are not very good either

Maybe @mr-lab share some more information about his research....

[mr-lab]

@mosvlad
we need a decoder that can process higher resolutions 512*3,072
@thuwzy is probably working on that .
Now we are working on an alternative ,Transfer CRM results to a 3d blob representing the shape of the subject ,
then remodel that blob into a model by moving vertex pos until they match target...still long way to see any good results .
CRM is the only True 3d generator , times and times again proven to provide consistent multi-view shots , no other project can do . will continue to prepare for a larger decoder .

[snowflakewang]

@thuwzy Hello, I am interested in upscaling the resolution of RGBs to get high-resolution textured meshes. You mentioned that you are working on 512-level decoders. I am curious about the maximum resolution that GPUs (maybe A100/A800) can handle. Is 1024 an acceptable resolution?