Some questions about Text-to-Texture task

[mdarhdarz]

Some questions about Text-to-Texture task

[Youwang-Kim]

Hi, thanks for your interest in our work.

1. Resolution & SD model

• We have used 512x512 for the UV maps. Indeed, increasing the resolution would result in better results. Also, we have used SD 2.1. However, using a better-performing text-to-image guidance model would improve the results.

2. SDS for texture generation

• Recent works (TEXTure, Text2Tex) use different methods than ours. They outpaint the mesh image using {Text, Depth}-conditioned image generation using ControlNet and back-project the generated RGB colors onto the mesh surface. This may generate good-quality textures.
• However, from our observation, their outpainting-based methods are hard to guarantee cross-view texture consistency. Since they outpaint the texture by increasing the range of view, subtle differences of generated textures in adjacent views tend to result in cross-view-inconsistent textures.
  
• On the other hand, we use randomly selected camera viewpoints and multi-view rendered images to optimize the texture, which can guide coherent multi-view texture generation.
• Furthermore, SDS can help disentangle the textures into PBR maps, resulting in better quality results + practical applications, e.g., relighting, and material control. Prior arts, such as TEXTure and Text2Tex, copy the generated image colors, which basically bakes the lighting effects, onto the mesh. With such formulation, the disentanglement of {diffuse, normal, specular, light} cannot be performed, which may limit the practical usage.

Hope this answers your questions.

[mdarhdarz]

Thanks for your efforts! I recomand Meshy (https://app.meshy.ai/zh/discover) and you can have a try. I'm concern about the inconsistency between industry and academia. Besides, I'm looking forward to your code so that I can add SDXL into it.

[Youwang-Kim]

Thanks for the comments and your recommendations.
Later, I'll try releasing the code with support for SDXL.