dunbar12138/pix2pix3d

pix2pix3D: Generating 3D Objects from 2D User Inputs

pix2pix3d's Issues

ResolvePackageNotFound cudatoolkit

I am having problem while installing packages with conda env create -f environment.yml

I tried several solutions like conda install pytorch==1.10 torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge but this is not working. Also I tried to find cudatoolkit 11.1 in anaconda.org and found this command: conda install -c zimmf cudatoolkit. So I installed it and tried again but still not working.

I am currently using macOS and could it be a problem?

Where pix2pix3d_seg2car.pkl

Where can I download checkpoints/pix2pix3d_seg2car.pkl, which converts a line drawing to a 3d car?
I can only find pix2pix3d_seg2cat.pkl.

When will the training code be released?

Thank you for your great work. When will the training code be released?

Code for real-time editing

Hi,

would you mind sharing the gui code for real-time editing? Thanks in advance!

Which edge style is using?

Thanks for sharing this amazing work.
Can I ask, in the edge-to-3D experiments, what style did you use with https://github.com/carolineec/informative-drawings
It has three styles in total: contour, anime, and opensketch.
Thanks

how to use "--input "?

Cudatoolkit 11.1

Does this repo can only be used uisng cudatoolkit 11.1?

I have installed cudatoolkit 11.1 but still getting an error.

How to obtain the segmentation maps for cats?

Hi, sorry for asking another question!
The paper says you used DINO features to cluster the cat face into segmentations.
Do you have the script for that?

could you release seg2face training model for test?

Hi, could you release seg2face training model for test?

what is the detail of limitations of second "dataset's distribution"

what is the actual limitation?

Can you provide the code for the evaluation model?

set '--no_eval=False ' to use metrics code will report many error.
like TypeError: cannot pickle '_io.BufferedReader' object

"How to solve this problem when encountered?"

UnicodeDecodeError: 'cp1' codec can't decode bytes in position 0--1: No mapping for the Unicode character exists in the target code page.

TypeError: TriPlaneSemanticEntangleGenerator.forward() missing 1 required positional argument: 'batch'

python train.py --outdir="./my_train_out" --cfg=afhq --data=data/afhq_v2_train_cat_512.zip --mask_data=data/afhqcat_seg_6c_v2.zip --data_type=seg --semantic_channels=6 --render_mask=True --dis_mask=True --neural_rendering_resolution_initial=64 --gpus=1 --batch=1 --mbstd-group=1 --gamma=5 --gen_pose_cond=True --random_c_prob=0.5 --lambda_d_semantic=0.1 --lambda_lpips=1 --lambda_cross_view=1e-4 --only_raw_recons=True --wandb_log=False --no_eval=False

when i use --no_eval=False ,Code throw exception,as follow:

  File "E:\python-eclipse-workspace-2022-12\pix2pix3D\metrics\metric_main.py", line 96, in kid50k_full
    kid = kernel_inception_distance.compute_kid(opts, max_real=1000000, num_gen=50000, num_subsets=100, max_subset_size=1000)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\python-eclipse-workspace-2022-12\pix2pix3D\metrics\kernel_inception_distance.py", line 29, in compute_kid
    gen_features = metric_utils.compute_feature_stats_for_generator(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\python-eclipse-workspace-2022-12\pix2pix3D\metrics\metric_utils.py", line 287, in compute_feature_stats_for_generator
    img = G(z=z, c=it_data, **opts.G_kwargs)['image']
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Python\Python3_11_3\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: TriPlaneSemanticEntangleGenerator.forward() missing 1 required positional argument: 'batch'

the G is TriPlaneSemanticEntangleGenerator

Training iterations and time

Hi, I want to know how many iterations do you use for training celeba with the pretrained eg3d weight. And how long is the training process？

Generalization of edge2car

Hi,
Thanks for releasing your code and model！
I tried to test new edge images but get similar results. This is the input,

and these are outputs

The generated video cannot align well with the input edge image.
The command is "python applications/generate_video.py --network checkpoints/pix2pix3d_edge2car.pkl --outdir examples --random_seed 1 --cfg edge2car --input examples/edge2car2.png". What is your suggestion for generating better results?
Best Wishes

404 while trying to download pre-trained models

Getting following output while running the download script for pre-trained models

(pix2pix3d) PS D:\ml\pix2pix3D> bash checkpoints/download_models.sh
checkpoints/download_models.sh: line 1: cd: $'checkpoints\r': No such file or directory
--2023-02-21 14:07:40--  http://cs.cmu.edu/~pix2pix3D/release.tar%0D
Resolving cs.cmu.edu (cs.cmu.edu)... 128.2.42.95, 128.2.1.30, 128.237.148.161
Connecting to cs.cmu.edu (cs.cmu.edu)|128.2.42.95|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.cs.cmu.edu/~pix2pix3D/release.tar%0d [following]
--2023-02-21 14:07:41--  http://www.cs.cmu.edu/~pix2pix3D/release.tar%0d
Resolving www.cs.cmu.edu (www.cs.cmu.edu)... 128.2.42.95, 128.237.148.168, 128.2.1.8
Reusing existing connection to cs.cmu.edu:80.
HTTP request sent, awaiting response... 404 Not Found
2023-02-21 14:07:41 ERROR 404: Not Found.

tar: release.tar\r: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
rm: cannot remove 'release.tar'$'\r': No such file or directory

about the discriminator

hi,thank you for your great work,
can i ask your discriminator architecture?It seems that you just use the reconstruction loss about image and mask to constrain the input and output mask shape,how about the reconstruction loss doesn't work well ?do you have any discriminator about the constrain between input and mask shape?

RuntimeError: CUDA error: operation not supported when calling `cusparseCreate(handle)`

Hi:), on my 4090 GPU, I had success implement environment.yml ,But Now I am having problem while generating samples with python applications/generate_samples.py --network checkpoints/pix2pix3d_seg2cat.pkl --outdir examples --random_seed 1 --cfg seg2cat --input_id 1666.

Setting up PyTorch plugin "bias_act_plugin"... Done.
Setting up PyTorch plugin "upfirdn2d_plugin"... Done.
Traceback (most recent call last):
  File "/home/ysh/NN01/pix2pix3D/applications/generate_samples.py", line 128, in <module>
    main()
  File "/home/ysh/NN01/pix2pix3D/applications/generate_samples.py", line 114, in main
    out = G.synthesis(ws, input_pose, noise_mode='const', neural_rendering_resolution=neural_rendering_resolution)
  File "<string>", line 946, in synthesis
  File "/home/ysh/.conda/envs/pix2pix3d/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ysh/NN01/pix2pix3D/./training/volumetric_rendering/renderer.py", line 109, in forward
    out = self.run_model(planes, decoder, sample_coordinates, sample_directions, rendering_options)
  File "/home/ysh/NN01/pix2pix3D/./training/volumetric_rendering/renderer.py", line 143, in run_model
    sampled_features = sample_from_planes(self.plane_axes, planes, sample_coordinates, padding_mode='zeros', box_warp=options['box_warp'])
  File "/home/ysh/NN01/pix2pix3D/./training/volumetric_rendering/renderer.py", line 63, in sample_from_planes
    projected_coordinates = project_onto_planes(plane_axes, coordinates).unsqueeze(1)
  File "/home/ysh/NN01/pix2pix3D/./training/volumetric_rendering/renderer.py", line 51, in project_onto_planes
    inv_planes = torch.linalg.inv(planes).unsqueeze(0).expand(N, -1, -1, -1).reshape(N*n_planes, 3, 3)
RuntimeError: CUDA error: operation not supported when calling `cusparseCreate(handle)`

安装过程出现这样的错误

Please release the training model

Thanks :)

P.S. I just hacked this AR AI app out for SIGGRAPH deadline yesterday. Would be cool to add pix2pix3d as a feature too! https://www.youtube.com/watch?v=KU9BHFbdABc

Reasoning does not proceed.

"Setting up PyTorch plugin "bias_act_plugin"..." I can't proceed from the notation "Setting up PyTorch plugin "bias_act_plugin"...". Do you know of any solution?

Edge2car training config.

Hi, really nice work! Could you please share the training script for Edge2car?

Generated mesh are so same

I have try different input_id to generate car mesh, such as id 222\888\1605\1607\1608, but the result are almost same.
Here is my command:
python applications/extract_mesh.py --network checkpoints/pix2pix3d_edge2car.pkl --outdir examples --cfg edge2car --input_id 1234
I have not changed any code. When I use your example pic, which is example_input_edge2car.png, the result is decent.

Can you give me some suggestion how to get different mesh when i use different input image?

Generating mesh from 2D sketch (edge2car)

I don't see how to use extract_mesh.py to provide a 2D sketch for edges as input. Could you please provide an example?

I'm thinking of integrating it to my XR scaffolding https://twitter.com/utopiah/status/1620089313592758274 based on your appendix C on broader impact.

Hello, can you publish seg2face training model for testing?

Hello, can you publish seg2face training model for testing?

raise AssertionError(f'Wrong size for dimension {idx}: got {size}, expected {ref_size}') AssertionError: Wrong size for dimension 1: got 0, expected 25

!python applications/generate_samples.py --network checkpoints/pix2pix3d_seg2cat.pkl --outdir examples --random_seed 1 --cfg seg2cat --input_id 2

was running in google colab

Traceback (most recent call last):
File "/content/pix2pix3D/applications/generate_samples.py", line 128, in
main()
File "/content/pix2pix3D/applications/generate_samples.py", line 113, in main
ws = G.mapping(z, input_pose, {'mask': input_label, 'pose': input_pose})
File "", line 919, in mapping
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "", line 361, in forward
File "/content/pix2pix3D/./torch_utils/misc.py", line 97, in assert_shape
raise AssertionError(f'Wrong size for dimension {idx}: got {size}, expected {ref_size}')
AssertionError: Wrong size for dimension 1: got 0, expected 25

raise AssertionError(f'Wrong size for dimension {idx}: got {size}, expected {ref_size}')
AssertionError: Wrong size for dimension 1: got 0, expected 25

Training stuck at the beginning

Thanks for this great works! I was trying to follow the training procedure, but it seems the training stucks at the beginning, it keeps showing the following for more than ten minutes and does not proceed any more:
`Training for 25000 kimg...

tick 0 kimg 0.0 time 1m 20s sec/tick 5.5 sec/kimg 1372.65 maintenance 74.8 cpumem 5.39 gpumem 21.01 reserved 22.00 augment 0.000`
By the way, I was trying to resume from "afhqcats512-128.pkl". Could any body give me some advice about how to move on?

About Hyperparameter

Thank you for your great work.

I would like to inquire about the settings of hyperparameters.

1. Which type of seg_weight do you use?(

   pix2pix3D/training/loss.py

   Line 338 in 63dc8d9

   if int(seg_weight) == 1:

   )
2. What are the learning rates of the G, D, D_semantic and Encoder？

Thank you.

Abnormal training results

Thank you for releasing the code. I train the code using the given celeba dataset. But I get abnormal training results.

I use the default script pix2pix3D/train_scripts/celeba_seg.sh.

The fakes0000012.jpg seems good.

But the fakes000080.jpg is abnormal.

Can you give me some advice? Thank you!

AssertionError: Wrong size for dimension 1: got 6, expected 8

How many image views are required for model training?
When running the training script, I get the following message:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 68, in _wrap
fn(i, *args)
File "/workspaces/3d_nlp/pix2pix3D/train.py", line 57, in subprocess_fn
training_loop.training_loop(rank=rank, **c)
File "/workspaces/3d_nlp/pix2pix3D/training/training_loop.py", line 331, in training_loop
misc.print_module_summary(D, [img, c])
File "/workspaces/3d_nlp/pix2pix3D/torch_utils/misc.py", line 230, in print_module_summary
outputs = module(*inputs)
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
result = forward_call(*args, **kwargs)
File "/workspaces/3d_nlp/pix2pix3D/training/dual_discriminator.py", line 165, in forward
x, img = block(x, img, **block_kwargs)
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
result = forward_call(*args, **kwargs)
File "/workspaces/3d_nlp/pix2pix3D/training/networks_stylegan2.py", line 623, in forward
misc.assert_shape(img, [None, self.img_channels, self.resolution, self.resolution])
File "/workspaces/3d_nlp/pix2pix3D/torch_utils/misc.py", line 97, in assert_shape
raise AssertionError(f'Wrong size for dimension {idx}: got {size}, expected {ref_size}')
AssertionError: Wrong size for dimension 1: got 6, expected 8

Could the issue stem from having only 6 image views in my dataset, or might there be other factors contributing to the problem?
Any assistance would be greatly appreciated.

Pretrained weights from EG3D

Thank you for your great work.

The paper says that 'we can significantly reduce the training time to 4 hours if we initialize parts of our model with pretrained weights from EG3D'. As I known, EG3D just has weights for FFHQ dataset. But pix2pix3D uses CelebaHQ-Mask dataset. So I want to know what weights do you use?

Thank you.

Google Colab

I would like a google colab

UnboundLocalError: local variable 'wandb' referenced before assignment

Good work!

When I use train_scripts/shapenet_edge.sh for training, I meet this bug:
Training for 25000 kimg...

tick 0 kimg 0.0 time 1m 29s sec/tick 8.1 sec/kimg 253.78 maintenance 81.2 cpumem 6.00 gpumem 19.10 reserved 21.52 augment 0.000
Traceback (most recent call last):
File "/userhome/pix2pix3D-main/train.py", line 540, in
main() # pylint: disable=no-value-for-parameter
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/userhome/pix2pix3D-main/train.py", line 534, in main
launch_training(c=c, desc=desc, outdir=opts.outdir, dry_run=opts.dry_run)
File "/userhome/pix2pix3D-main/train.py", line 112, in launch_training
torch.multiprocessing.spawn(
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
while not context.join():
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 160, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/opt/conda/envs/pix2pix3d/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
fn(i, *args)
File "/userhome/pix2pix3D-main/train.py", line 57, in subprocess_fn
training_loop.training_loop(rank=rank, **c)
File "/userhome/pix2pix3D-main/training/training_loop.py", line 690, in training_loop
log_table(G_ema, grid_z, grid_i, grid_c, grid_m, grid_p, mask_type=training_set_kwargs.data_type, global_step=cur_nimg//1000, device=device, wandb=wandb)
UnboundLocalError: local variable 'wandb' referenced before assignment

Have you ever meet this wrong? Could you please tell me the possible reason? Thank you very much!

can we use a 2d image of a furniture and turn it into a 3d model?

Question about implementation of triplane projection

Hi, thanks for releasing the code.

I found that tri-plane projection of the code is the same as the original implementation, which turned out to be an inaccurate implementation.

The projection code projects 3D coordinates to XY, XZ, ZX planes not XY, YZ, ZX planes.

Please refer to this issue.

I wonder if the projection of 3D coordinates onto these XY, XZ, and ZX planes is what you intended in your paper.

Thank you:)