Training

[shqmffl486]

Training

[zerchen]

Hi,

I created this split only for students to conduct experiments under limited computing resources. I did not do experiments using the sdf_hand_mini and sdf_obj_mini.
To use this split, you need to generate a new json file like this and use it in your config file (https://github.com/zerchen/AlignSDF/blob/master/experiments/obman/30k_1e2d_mlp5.json).
Hope it helps.

[shqmffl486]

Thank you for your reply.
I learned it beforehand and trained it, but there seems to be an error in the process of making the last mesh.
What do you think is the problem?
So the Eval_obman file and several files were created in it, but the contents were missing

DeepSdf - INFO - time used: 85.93944382667542
DeepSdf - INFO - save at 100
DeepSdf - INFO - Distributing BatchNorm running means and vars
Traceback (most recent call last):
File "train.py", line 715, in
main_function(exp_cfg, args.continue_from, args.local_rank, args.opt_level, args.slurm)
File "train.py", line 669, in main_function
reconstruct(encoderDecoder, specs, split_filename, output_path, start_point=start_points[local_rank], end_point=end_points[local_rank], task=task, device=device, cube_dim=128, label_out=use_optim_mano, eval_mode=use_eval_mode)
File "/home/gaeun/mount4t/AlignSDF/reconstruct.py", line 95, in reconstruct
utils.mesh.create_mesh_combined_decoder(hand_branch, obj_branch, cls_branch, loaded_model.module.decoder, latent, mano_results, obj_results, cam_intr, specs, mesh_filename, N=cube_dim, max_batch=int(2 ** 18), scale=scale, device=device, label_out=label_out, viz=viz, eval_mode=eval_mode, task=task)
File "/home/gaeun/mount4t/AlignSDF/utils/mesh.py", line 157, in create_mesh_combined_decoder
out_labels[head: min(head + max_batch, num_out_vertices)] = predicted_class.argmax(dim=1).detach().cpu()
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
Killing subprocess 19631
Traceback (most recent call last):
File "/home/gaeun/anaconda3/envs/alignsdf/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/gaeun/anaconda3/envs/alignsdf/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/gaeun/anaconda3/envs/alignsdf/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in
main()
File "/home/gaeun/anaconda3/envs/alignsdf/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/home/gaeun/anaconda3/envs/alignsdf/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/gaeun/anaconda3/envs/alignsdf/bin/python', '-u', 'train.py', '--local_rank=0', '-e', 'experiments/obman/30k_1e2d_mlp5.json']' returned non-zero exit status 1.

[shqmffl486]

I think this line is the problem.
mesh.py 157, out_labels[head: min(head + max_batch, num_out_vertices)] = predicted_class.argmax(dim=1).detach().cpu()