nvlabs / nvae Goto Github PK
View Code? Open in Web Editor NEWThe Official PyTorch Implementation of "NVAE: A Deep Hierarchical Variational Autoencoder" (NeurIPS 2020 spotlight paper)
Home Page: https://arxiv.org/abs/2007.03898
License: Other
The Official PyTorch Implementation of "NVAE: A Deep Hierarchical Variational Autoencoder" (NeurIPS 2020 spotlight paper)
Home Page: https://arxiv.org/abs/2007.03898
License: Other
First of all I would like to say thank you for the great VAE implementation. Looking forward to Celeb256 training instructions!
I tried preprocessing CelebA64 but got an error when executing create_celeba64_lmdb.py. The way you obtain the dataset (dset.celeba.CelebA) downloads corrupted img_align_celeba.zip, I was able to fix the problem by replacing 'dset.celeba.CelebA' with 'dset.CelebA'. According to the official API this is correct way to do it (https://pytorch.org/docs/stable/torchvision/datasets.html#celeba).
I'd like to train the model on my own dataset.but i don't know how to use the code to train it. do i need to read the code carefully to modify the code to train it. or if i can just use the config file or command without to read the detailed code? time is really tight so I perfer if someone can help me answer this question.
When i run the folllowing command to store the CelebA-HQ 256x256 dataset in an LMDB dataset,
python convert_tfrecord_to_lmdb.py --dataset=celeba --tfr_path=$DATA_DIR/celeba/celeba-tfr --lmdb_path=$DATA_DIR/celeba/celeba-lmdb --split=train
I am getting the following error:
Traceback (most recent call last): File "convert_tfrecord_to_lmdb.py", line 12, in <module> from tfrecord.torch.dataset import TFRecordDataset ModuleNotFoundError: No module named 'tfrecord'
I found only the following github issue relevant to the query: vahidk/tfrecord#1
I request you to please guide me.
Hi,
I am trying to run NVAE on my machine with your command line for CIFAR10 (updating only the .. from 8 to 4 cause I own 4 GPUs):
export EXPR_ID=/home/dsi/eyalbetzalel/NVAE/logs
export DATA_DIR=/home/dsi/eyalbetzalel/NVAE/data
export CHECKPOINT_DIR=/home/dsi/eyalbetzalel/NVAE/cpt
export CODE_DIR=/home/dsi/eyalbetzalel/NVAE
cd $CODE_DIR
nohup python train.py --data $DATA_DIR/cifar10 --root $CHECKPOINT_DIR --save $EXPR_ID --dataset cifar10 \
--num_channels_enc 128 --num_channels_dec 128 --epochs 400 --num_postprocess_cells 2 --num_preprocess_cells 2 \
--num_latent_scales 1 --num_latent_per_group 20 --num_cell_per_cond_enc 2 --num_cell_per_cond_dec 2 \
--num_preprocess_blocks 1 --num_postprocess_blocks 1 --num_groups_per_scale 30 --batch_size 32 \
--weight_decay_norm 1e-2 --num_nf 1 --num_process_per_node 4 --use_se --res_dist --fast_adamax &> NVAE_DSIGPU13_test_2_22102020.out &
and get this error:
File "/home/dsi/eyalbetzalel/miniconda3/envs/NVAE_env/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/dsi/eyalbetzalel/miniconda3/envs/NVAE_env/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "train.py", line 281, in init_processes
fn(args)
File "train.py", line 92, in main
train_nelbo, global_step = train(train_queue, model, cnn_optimizer, grad_scalar, global_step, warmup_iters, writer, logging)
File "train.py", line 160, in train
utils.average_params(model.parameters(), args.distributed)
File "/home/dsi/eyalbetzalel/NVAE/utils.py", line 274, in average_params
dist.all_reduce(param.data, op=dist.ReduceOp.SUM)
File "/home/dsi/eyalbetzalel/miniconda3/envs/NVAE_env/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 936, in all_reduce
work = _default_pg.allreduce([tensor], opts)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:518, unhandled system error, NCCL version 2.4.8
am I doing something wrong?
Thanks,
Eyal
@
I think this work is likely to win the outstanding paper award. I am looking forward to the oral presentation.
First of all, the code works out-of-the-box. I am currently training on MNIST and the results are reasonable. I do have a couple of questions, which I think would benefit others as well. Some of the questions might be trivial, given that I am not an expert in VAEs.
From the paper, Figure 2 (caption): "... and h is a trainable parameter":
The description of h does not appear again in the paper (I might have missed it). What is it? Does it correspond to self.prior_ftr0 in line147?
What are the magic numbers (multiplication and subtraction) in line 334? Is this basically transforming the intensity range [0.0, 1.0] to [-1.0, 1.0]?
Do combiner_enc cells in line 344 correspond to the red ⊕ symbols along the encoder path in the in Figure 2?
What does the enc0 function (or equivalently ftr) in line 355 represent? Is this the initial diamond residual layer that immediately follows x in Figure 2?
What is the function of pre-processing layers and more precisely down_pre and normal_pre in the init_pre_process function in line 201? I can tell they have something to do with the bottom-up and top-down paths in Figure 2 but I am not sure what they do.
Similar question to 5 regarding "post-processing" layers. What is their function? Why do you use them?
Would it be fair to say that the latent representation of an image using the encoder network would be the mean of each residual normal distribution? If that is the case, would this essentially be mu_q in line 394 (corresponding to z1 in the decoder path) and z in line 396 for z2 and subsequent latents?
Thank you for sharing your code. I wish you the best.
Hello Dr. Vahdat,
This is indeed impressive work !!! However I am struggling with the training process.
Using Pytorch 1.6.0 with cuda 10.1
Training using 4 (Not V-100) GPUs of size ~12GB each. Reduced batch size to 8 to fit memory. No other changes apart from this. Followed the instructions exactly as given in Readme. But the training logs show that I am get "nan" losses
Is there any other pre-processing step I need to do for the dataset? Perhaps any other minor detail which you felt was irrelevant to mention in the readme? Any help you can provide is greatly appreciated.
Hello, great work! I am trying to understand NVAE, but I could not figure out the role of self.prior_ftr0 or h block in Fig2 in the decoder model. Correct me if im wrong, but i could not find any details regarding h in the paper; and the code defines it to be a random vector generated in the initialization of the model and then being used in the forward pass to kickstart the decoder model. my question is why did you chose this design rather than starting with just z1? Is there some part that Im missing here?
thank you so much for the great paper and repo.
Thank you for sharing the implementation of the DDGAN model. I am trying to train the model on FFHQ 256x256 dataset. I used the NVLabs/NVAE repository for the dataset preparation. I have the file structure as follows:
To use another dataset similar to the CelebA-HQ 256x256, I modified the train
function given in the line 190 of the train_ddgan.py
file.
elif args.dataset == 'ffhq_256':
train_transform = transforms.Compose([
transforms.Resize(args.image_size),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))
])
dataset = LMDBDataset(root='/datasets/ffhq-lmdb/', name='ffhq', train=True, transform=train_transform)
My implementation for the DDGAN uses 4 NVIDIA GTX 1080ti GPUs with a total batch size of 32 for training the CelebA-HQ 256x256 dataset
(--batch_size 8
and --num_process_per_node 4
)
I use the following command for training!python3 train_ddgan.py --dataset ffhq_256 --image_size 256 --exp ddgan_celebahq_exp1 --num_channels 3 --num_channels_dae 64 --ch_mult 1 1 2 2 4 4 --num_timesteps 2 --num_res_blocks 2 --batch_size 8 --num_epoch 800 --ngf 64 --embedding_type positional --use_ema --r1_gamma 2. --z_emb_dim 256 --lr_d 1e-4 --lr_g 2e-4 --lazy_reg 10 --num_process_per_node 4 --save_content
I am getting the following output message:
Node rank 0, local proc 0, global proc 0
Node rank 0, local proc 1, global proc 1
Node rank 0, local proc 2, global proc 2
Node rank 0, local proc 3, global proc 3
Process Process-4:
Traceback (most recent call last):
File "/usr/local/apps/python-3.8.3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/apps/python-3.8.3/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "train_ddgan.py", line 482, in init_processes
fn(rank, gpu, args)
File "train_ddgan.py", line 248, in train
dataset = LMDBDataset(root='/datasets/ffhq-lmdb/', name='ffhq', train=True, transform=train_transform)
File "/home/manisha.padala/gan/denoising-diffusion-gan/datasets_prep/lmdb_datasets.py", line 33, in __init__
self.data_lmdb = lmdb.open(lmdb_path, readonly=True, max_readers=1,
lmdb.Error: /datasets/ffhq-lmdb/train.lmdb: No such file or directory
Process Process-2:
Traceback (most recent call last):
File "/usr/local/apps/python-3.8.3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/apps/python-3.8.3/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "train_ddgan.py", line 482, in init_processes
fn(rank, gpu, args)
File "train_ddgan.py", line 248, in train
dataset = LMDBDataset(root='/datasets/ffhq-lmdb/', name='ffhq', train=True, transform=train_transform)
File "/home/manisha.padala/gan/denoising-diffusion-gan/datasets_prep/lmdb_datasets.py", line 33, in __init__
self.data_lmdb = lmdb.open(lmdb_path, readonly=True, max_readers=1,
lmdb.Error: /datasets/ffhq-lmdb/train.lmdb: No such file or directory
Process Process-1:
Traceback (most recent call last):
File "/usr/local/apps/python-3.8.3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/apps/python-3.8.3/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "train_ddgan.py", line 482, in init_processes
fn(rank, gpu, args)
File "train_ddgan.py", line 248, in train
dataset = LMDBDataset(root='/datasets/ffhq-lmdb/', name='ffhq', train=True, transform=train_transform)
File "/home/manisha.padala/gan/denoising-diffusion-gan/datasets_prep/lmdb_datasets.py", line 33, in __init__
self.data_lmdb = lmdb.open(lmdb_path, readonly=True, max_readers=1,
lmdb.Error: /datasets/ffhq-lmdb/train.lmdb: No such file or directory
Process Process-3:
Traceback (most recent call last):
File "/usr/local/apps/python-3.8.3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/apps/python-3.8.3/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "train_ddgan.py", line 482, in init_processes
fn(rank, gpu, args)
File "train_ddgan.py", line 248, in train
dataset = LMDBDataset(root='/datasets/ffhq-lmdb/', name='ffhq', train=True, transform=train_transform)
File "/home/manisha.padala/gan/denoising-diffusion-gan/datasets_prep/lmdb_datasets.py", line 33, in __init__
self.data_lmdb = lmdb.open(lmdb_path, readonly=True, max_readers=1,
lmdb.Error: /datasets/ffhq-lmdb/train.lmdb: No such file or directory
Arash Vahdat and Jan Kautz, thank you for a great paper and the code that you are providing!
My GPUs: 8 V100s 32GB.
Out of GPU Memory.
I would like to point out that provided command for training on CelebaA HQ 256 runs out of GPU memory whether it is default command with reduced amount of GPUs (24 to 8) batch_size 4 or suggested one for 8 GPUs with batch size 6. With reduced amount of GPUs maintaining 30 channels, batch_size of 3 works. Command:
python train.py --data /celeba/celeba-lmdb --root /NVAE/checkpoints --save 1 --dataset celeba_256 --num_channels_enc 30 --num_channels_dec 30 --epochs 300 --num_postprocess_cells 2 --num_preprocess_cells 2 --num_latent_scales 5 --num_latent_per_group 20 --num_cell_per_cond_enc 2 --num_cell_per_cond_dec 2 --num_preprocess_blocks 1 --num_postprocess_blocks 1 --weight_decay_norm 1e-2 --num_groups_per_scale 16 --batch_size 3 --num_nf 2 --ada_groups --min_groups_per_scale 4 --weight_decay_norm_anneal --weight_decay_norm_init 1. --num_process_per_node 8 --use_se --res_dist --fast_adamax --num_x_bits 5 --cont_training
The command suggested for 8 GPUs with 24 channels works with batch_size of 5. Command:
python train.py --data /celeba/celeba-lmdb --root /NVAE/checkpoints --save 1 --dataset celeba_256 --num_channels_enc 24 --num_channels_dec 24 --epochs 300 --num_postprocess_cells 2 --num_preprocess_cells 2 --num_latent_scales 5 --num_latent_per_group 20 --num_cell_per_cond_enc 2 --num_cell_per_cond_dec 2 --num_preprocess_blocks 1 --num_postprocess_blocks 1 --weight_decay_norm 1e-2 --num_groups_per_scale 16 --batch_size 5 --num_nf 2 --ada_groups --min_groups_per_scale 4 --weight_decay_norm_anneal --weight_decay_norm_init 1. --num_process_per_node 8 --use_se --res_dist --fast_adamax --num_x_bits 5
So -1 on batch_size in both cases, otherwise out of GPU memory.
NaN
I made 4 runs overall with 2 suggested commands, rerunning every command twice, but all of them didnt even reach epoch 100. "NaN or Inf found in input tensor" was encountered, sometimes training breaks at epoch 35, sometimes at 70. Starting from last checkpoint goes nowhere, same problem. Problem is listed in "Known Issues", you are mentioning: "commands above can be trained in a stable way", in my case given commands were unstable, the only difference is a reduced batch_size by one, I doubt reducing it by one can make such a big difference.
Did anyone else encounter these issues?
Ill play around with listed tricks to stabilize the training and report if something will remedy the NaN.
Thanks!
Thanks for this implementation!
I think the execution script for cifar-10 provided in the readme has an extra --save
in it and should read:
export EXPR_ID=UNIQUE_EXPR_ID export DATA_DIR=PATH_TO_DATA_DIR export CHECKPOINT_DIR=PATH_TO_CHECKPOINT_DIR export CODE_DIR=PATH_TO_CODE_DIR cd $CODE_DIR python train.py --data $DATA_DIR/cifar10 --root $CHECKPOINT_DIR --save $EXPR_ID --dataset cifar10 \ --num_channels_enc 128 --num_channels_dec 128 --epochs 400 --num_postprocess_cells 2 --num_preprocess_cells 2 \ --num_latent_scales 1 --num_latent_per_group 20 --num_cell_per_cond_enc 2 --num_cell_per_cond_dec 2 \ --num_preprocess_blocks 1 --num_postprocess_blocks 1 --num_groups_per_scale 30 --batch_size 32 \ --weight_decay_norm 1e-2 --num_nf 1 --num_process_per_node 8 --use_se --res_dist --fast_adamax
It crashes as is right now.
Hi!
I would like to explore the latent variables for given images, in order to cluster images based on their latent space representations.
Which output do I have to take to get the variables?
Do I have to take z
after applying normalizing flows, so around model.py#L366?
Thanks for your help!
Marc
PS: Great paper, and thanks for the helpful reference implementation!
Can you please release checkpoint for ImageNet NVAE? Thanks!
Hi there, I am just confused by the inverse autoregressive flow. Do you use another network model to fit the distribution q(z|x)? Can I understand in the following way?
As far as my know, in the flow based model, people want to model the data distribution p(x) , so from a random z, we can get x=f(z). Here in this paper , q(z|x) is your data distribution you will model, you train another network g, again, from a random e~N(0,I), you get a z=g(e), then you can sample a realistic image through the decoder of the NVAE model, Decoder(z).
Line 213 in 38eb997
* total_kl
in this line is redundant because in the next line kl_coeff_i
is divided by its mean.mean(abs(kl_i))
is used as a weight factor for kl_i
? / alpha_i
equivalent to * num_group / feature_resolution
?Thank you for your sharing
I want to know that how to get Latent variables and save it
It's really important to me.
Hi and thanks for sharing the code.
I am doing some tests with medical images, 16-bit and one channel input, therefore employing the NormalDecoder
.
The implementation takes as optional parameter num_bits
but is not used as for the DiscMixLogistic
. Is the implementation complete? or is just included for some kind of legacy?
Thanks in advance!
Great work!
I have observed that some generated images by official checkpoint of CelebA64 are NaN . This might be caused by "Batch Norm Statistics Readjusted" introduced by the paper, becuase without BN statistics readjusted, none of the generated images are NaN (but the generated iamges quality have a sharp decline). Do you have this problem and how to deal with it?
Thank you for the excelllent paper and the great repo!
Hi, thank you for your outstanding work in making VAEs great again !
My question is about the pre processing of Gaussian parameters in distributions.py:
def soft_clamp5(x: torch.Tensor):
return x.div_(5.).tanh_().mul(5.) # 5. * torch.tanh(x / 5.) <--> soft differentiable clamp between [-5, 5]
[...]
self.mu = soft_clamp5(mu)
log_sigma = soft_clamp5(log_sigma)
self.sigma = torch.exp(log_sigma) + 1e-2
I don't think this is discussed in the paper, what is the role of this pre processing ?
It seems to be linked with the model's stability when I remove it. Do you have results on the relationship between this and the other stabilization methods discussed in the paper ?
Thank you
I'm quite confused about FID of CelebA-HQ.
In NCP-VAE and VAEBM paper, it is reported as 40.26, while recent paper LSGM reported as 29.76.
Were there further improvement of NVAE after publication of NCP-VAE and VAEBM?
Hi, Thanks a lot for the nice-documented repo. I have a question regarding traversing the latent space.
For performing the traversing experiment, do I only need to modify mu_q
in line 363
in the file model.py
? For example, replacing one of the columns in mu_q
with values between [-3,3]. Or do I also need to consider the values in self. enc_sampler
?
I really appreciate any help you can provide.
Thanks for sharing your code. I have found that the hyper-parameter 'num_latent_scales' on ImageNet 32x32 and CIFAR-10 has been set to be 1, which is quite smaller than that of 5 on FFHQ and CelebA HQ dataset. What will happen if we set this number as 5 on ImageNet 32x32 dataset ? Thanks~
Hi there, I was wondering if I can ask a question and get a bit of help about the representation power of NVAE. I would really appreciate it.
For one scenario, if I have two people, A, and B, and I get their representations, Z_a, and Z_b (all 25 Z's for 3 scales), I can somewhat change from person A to person B, by doing an interpolation such that new Z_a = (1-alpha) * Z_a + (alpha) * Z_b and decode, though the representation is not too smooth and alpha needs to be high (0.8) to see a change. I was wondering if the authors have tried something like this and see something similar?
Now, I only have Z_a, encoded from a person A. But unfortunately, I cannot modify Z_a by any means, to get a different person C, no matter what I add to the Z_a. If I add a large direction to it, I get a grainier/noisier image, but not a new person C.
I have also tried disabling flows but no luck. I feel that perhaps the mu and sigma coming from the encoder is too strong so that I cannot change the Z to get a new person (bc there's a part where NVAE concat the encoding and decoding mu,sigma). But then again I'm not sure.
Last point. in sampling, I see that we generate a new z_0 by sampling from N(0,1), and after going through the hierarchical Guassian, we get a nice new image. But let say I replace the z_0 by the z_0 of the encoded Z's of the person A, i get a junky/bad/trippy image, so I am at a loss bc the z_0 of a person A going through the hierarchical NVAE does not yield a person image.
Hey, I am trying to train your https://github.com/NVlabs/denoising-diffusion-gan Denoising Diffusion GAN model and my image size is 256. However, the problem is that repository recommends this repository to download and extract celeb as a lmdb file. However, when I try to run the script as it is described, I am getting the following error.
Can you fix the problem. I can see that tfrecords repository has been archived but may be the problem is different. Thanks a lot.
Hi thanks for the great work!
I am just wondering do you have plans for releasing the FID scores on varying datasets for NVAE?
Thanks!
Seems a parameter is missing in the "torch.batch_norm_backward_elemt" function. Could anyone tell me how to solve this problem?
I am trying to play with the results of the model with the pretrained checkpoints, however, I continue to get out of memory errors when passing even a single image through the model for inference.
I load the state_dict
and args
of the FFHQ pretrained checkpoint into the model and pass in a single image of shape (1,3,256,256)
, dtype torch.float32
, with values between [0, 1]
(per the LMDB DataLoader) into the model's forward
pass. I am using a Tesla V100 16GB, torch==1.6.0
, CUDA 11.0, and Ubuntu 18.04.
I did a bit of debugging -- it looks like this Out of Memory error occurs when passing the image through the loop over the dec_tower
.
Is there any reason, algorithmically, that the information for a single 256x256 image should consume more 16GB of GPU memory? Is there a memory leak somewhere?
Any insight and guidance on how to fix this would be appreciated.
Thanks for your interesting works.
I wonder how to generate the interpolations by traversing in the latent space.
Could you provide the source code about the latent space traverse if convenient?
Thanks.
Can we run this model without using distributed nature of the code? I simply want to run the code for 1024x1024 images without doing any parallelization? Can't believe I am striggnling with this for a week now.....
Thank you team for the sharing the project resources. I am trying to process the CelebA-HQ 256x256 dataset for the DDGAN model. The DDGAN repository recommends going over the dataset preparation methods in the NVAE repository (this repository).
The following commands will download tfrecord files from GLOW and convert them to store them in an LMDB dataset.
Use the link by openai/glow for downloading the CelebA-HQ 256x256 dataset (4 Gb).
To convert/store the CelebA-HQ 256x256 dataset to/as the lmdb dataset one needs to install module called "tfrecord".
The missing module error can be rectified by simply executing the command pip install tfrecord
.
!mkdir -p $DATA_DIR/celeba
%cd $DATA_DIR/celeba
!wget https://openaipublic.azureedge.net/glow-demo/data/celeba-tfr.tar
!tar -xvf celeba-tfr.tar
%cd $CODE_DIR/scripts
!pip install tfrecord
!python convert_tfrecord_to_lmdb.py --dataset=celeba --tfr_path=$DATA_DIR/celeba/celeba-tfr --lmdb_path=$DATA_DIR/celeba/celeba-lmdb --split=train
The final command !python convert_tfrecord_to_lmdb.py --dataset=celeba --tfr_path=$DATA_DIR/celeba/celeba-tfr --lmdb_path=$DATA_DIR/celeba/celeba-lmdb --split=train
gives the following output:
.
.
.
26300
26400
26500
26600
26700
26800
26900
27000
added 27000 items to the LMDB dataset.
Traceback (most recent call last):
File "convert_tfrecord_to_lmdb.py", line 73, in <module>
main(args.dataset, args.split, args.tfr_path, args.lmdb_path)
File "convert_tfrecord_to_lmdb.py", line 58, in main
print('added %d items to the LMDB dataset.' % count)
lmdb.Error: mdb_txn_commit: Disk quota exceeded
I am not sure I have made the LMDB dataset properly, I request you to guide me.
Thanks for the interesting work.
I want to reproduce the NVAE model in CelebAHQ 256 dataset. Could you provide the main NVAE training scripts with detailed parameters in the Celeba-HQ dataset?
Thanks for your help.
As per the readme file, I have changed some values in the arguments to make the model easier to train. Since, I have only one gpu, I have change num_process_per_node to 1.
But I am getting this error.
python train.py --data D:\Project\TAMU\Hierarchical_Variational_Autoencoders\NVAE#start-of-content\mnist --root D:\Project\TAMU\Hierarchical_Variational_Autoencoders\NVAE#start-of-content\CHECKPOINT\ --save 01 --dataset mnist --batch_size 200 --epochs 100 --num_latent_scales 2 --num_groups_per_scale 10 --num_postprocess_cells 3 --num_preprocess_cells 3 --num_cell_per_cond_enc 1 --num_cell_per_cond_dec 1 --num_latent_per_group 20 --num_preprocess_blocks 2 --num_postprocess_blocks 2 --weight_decay_norm 1e-2 --num_channels_enc 16 --num_channels_dec 16 --num_nf 0 --ada_groups --num_process_per_node 1 --use_se --res_dist --fast_adamax
Experiment dir : D:\Project\TAMU\Hierarchical_Variational_Autoencoders\NVAE#start-of-content\CHECKPOINT/eval-01
starting in debug mode
Traceback (most recent call last):
File "train.py", line 415, in
init_processes(0, size, main, args)
File "train.py", line 280, in init_processes
dist.init_process_group(backend='nccl', init_method='env://', rank=rank, world_size=size)
File "D:\ProgramData\Anaconda3\lib\site-packages\torch\distributed\distributed_c10d.py", line 421, in init_process_group
init_method, rank, world_size, timeout=timeout
File "D:\ProgramData\Anaconda3\lib\site-packages\torch\distributed\rendezvous.py", line 82, in rendezvous
raise RuntimeError("No rendezvous handler for {}://".format(result.scheme))
RuntimeError: No rendezvous handler for env://
I am new to PyTorch too, used Tensorflow till now.
Can you please tell me what type of error is this and how can I solve it?
Hi there, thank you for the great job!
I noticed that your implementation has an argument called arch_instance
, set to default as res_mbconv
. From the provided readme, it appears that this parameter is set as default in all your runs.
My question is: is this just an old argument used for testing different architectures? When training a new model, is it correct to leave it as default?
Can you also please provide some more info on what this argument does/what are the different tried architectures?
Thank you
Update:
--num_process_per_node 8 denotes the gpu number !!! So i need to change it to 1.
The first silly question asked under a great project, just laughed, I hope everyone succeeds
Hi, dear Arash Vahdat ,
NVAE is a great job! We are excited to meet this official implement.
After hesitating for two days, I still can’t help but ask, any friend else has successfully reproduce this implement in private machine?
During my running process, there still some errors, and I'm not sure whether this is purely resulted from my gpu.
Here is the traceback message:
(hsj-torch-gpu16) hsj@hsj:/data/hsj/NVAE$ python train.py --data ./scripts/data1/datasets/celeba_org/celeba64_lmdb --root ./CHECKPOINT_DIR --save ./EXPR_ID --dataset celeba_64 --num_channels_enc 32 --num_channels_dec 32 --epochs 90 --num_postprocess_cells 2 --num_preprocess_cells 2 --num_latent_scales 3 --num_latent_per_group 20 --num_cell_per_cond_enc 1 --num_cell_per_cond_dec 1 --num_preprocess_blocks 1 --num_postprocess_blocks 1 --weight_decay_norm 1e-1 --num_groups_per_scale 5 --batch_size 1 --num_nf 1 --ada_groups --num_process_per_node 8 --use_se --res_dist --fast_adamax Experiment dir : ./CHECKPOINT_DIR/eval-./EXPR_ID Node rank 0, local proc 0, global proc 0 Node rank 0, local proc 1, global proc 1 Node rank 0, local proc 2, global proc 2 Node rank 0, local proc 3, global proc 3 Node rank 0, local proc 4, global proc 4 Node rank 0, local proc 5, global proc 5 Node rank 0, local proc 6, global proc 6 Node rank 0, local proc 7, global proc 7 Process Process-1: Traceback (most recent call last): File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 280, in init_processes dist.init_process_group(backend='nccl', init_method='env://', rank=rank, world_size=size) File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 422, in init_process_group store, rank, world_size = next(rendezvous_iterator) File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 172, in _env_rendezvous_handler store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout) RuntimeError: Address already in use THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal Process Process-3: Traceback (most recent call last): File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 279, in init_processes torch.cuda.set_device(args.local_rank) File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/site-packages/torch/cuda/__init__.py", line 281, in set_device torch._C._cuda_setDevice(device) RuntimeError: cuda runtime error (101) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:59 THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal Process Process-4: Traceback (most recent call last): File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 279, in init_processes torch.cuda.set_device(args.local_rank) File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/site-packages/torch/cuda/__init__.py", line 281, in set_device torch._C._cuda_setDevice(device) RuntimeError: cuda runtime error (101) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:59 THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal Process Process-5: Traceback (most recent call last): File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 279, in init_processes torch.cuda.set_device(args.local_rank) File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/site-packages/torch/cuda/__init__.py", line 281, in set_device torch._C._cuda_setDevice(device) RuntimeError: cuda runtime error (101) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:59 THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal Process Process-6: Traceback (most recent call last): File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 279, in init_processes torch.cuda.set_device(args.local_rank) File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/site-packages/torch/cuda/__init__.py", line 281, in set_device torch._C._cuda_setDevice(device) RuntimeError: cuda runtime error (101) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:59 THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal Process Process-8: Process Process-7: Traceback (most recent call last): File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 279, in init_processes torch.cuda.set_device(args.local_rank) File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/site-packages/torch/cuda/__init__.py", line 281, in set_device torch._C._cuda_setDevice(device) RuntimeError: cuda runtime error (101) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:59 Traceback (most recent call last): File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 279, in init_processes torch.cuda.set_device(args.local_rank) File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/site-packages/torch/cuda/__init__.py", line 281, in set_device torch._C._cuda_setDevice(device) RuntimeError: cuda runtime error (101) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:59 Process Process-2: Traceback (most recent call last): File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/bjfu/anaconda3/envs/hsj-torch-gpu16/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 281, in init_processes fn(args) File "train.py", line 42, in main model = AutoEncoder(args, writer, arch_instance) File "/data/hsj/NVAE/model.py", line 163, in __init__ self.init_normal_sampler(mult) File "/data/hsj/NVAE/model.py", line 270, in init_normal_sampler nf_cells.append(PairedCellAR(self.num_latent_per_group, num_c1, num_c2, arch)) File "/data/hsj/NVAE/model.py", line 93, in __init__ self.cell1 = CellAR(num_z, num_ftr, num_c, arch, mirror=False) File "/data/hsj/NVAE/model.py", line 66, in __init__ self.conv = ARInvertedResidual(num_z, num_ftr, ex=ex, mirror=mirror) File "/data/hsj/NVAE/neural_ar_operations.py", line 147, in __init__ layers.extend([ARConv2d(inz, hidden_dim, kernel_size=3, padding=1, masked=True, mirror=mirror, zero_diag=True), File "/data/hsj/NVAE/neural_ar_operations.py", line 87, in __init__ self.mask = torch.from_numpy(create_conv_mask(kernel_size, C_in, groups, C_out, zero_diag, mirror)).cuda() RuntimeError: CUDA error: out of memory (hsj-torch-gpu16) bjfu@bjfu-15043:/data/hsj/NVAE$ lspci -vnn | grep -A6 "VGA" File "/data/hsj/NVAE/neural_ar_operations.py", line 147, in __init__ layers.extend([ARConv2d(inz, hidden_dim, kernel_size=3, padding=1, masked=True, mirror=mirror, zero_diag=True), File "/data/hsj/NVAE/neural_ar_operations.py", line 87, in __init__ self.mask = torch.from_numpy(create_conv_mask(kernel_size, C_in, groups, C_out, zero_diag, mirror)).cuda() RuntimeError: CUDA error: out of memory
Ps. Due to the GoogleDrive problem, I download the data specifically, and added into /data1 file ,then convert them into lmdb type data.
My first instinct is that the gpu memory is not enough, so I have reduced the batch parameters and model parameters as much as possible, as you can see in the first command line, however, it still don't work.
Ps. Devices Information:
`
NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2
2 * 1080ti ,while second one is running other job, only device 0 available
torch==1.6.0
torchvision==0.7.0
`
This NVAE is a great breakthrough job, expecting that we all can reproduce this job and get more inspirations from this.
Looking forward to get any useful suggestion,
Sincerely
Luke Huang
Thank you sir for sharing the scripts for dataset preparation. I am trying to implement the DDGAN model on the FFHQ 256x256 dataset. I have used the FFHQ 256x256 resized dataset from the kaggle since the FFHQ 1024x1024 dataset has a size of 90 GB, which exceeds the limits of my resources.
The Kaggle dataset has the files in archive.zip file, which has a directory "resized" which contains the 70k .jpg files.
The file structure is as follows:
archive.zip
├ resized
├ (70k images)
I am using google drive and colab notebooks for the implementation. I am using the file setup with CODE_DIR = "/content/drive/MyDrive/Repositories/NVAE" and DATA_DIR = "/content/drive/MyDrive/Repositories/NVAE/dataset_nvae"
. When I try to run the command !python create_ffhq_lmdb.py --ffhq_img_path=$DATA_DIR/ffhq/resized/ --ffhq_lmdb_path=$DATA_DIR/ffhq/ffhq-lmdb --split=train
, I get the following error message:
Traceback (most recent call last):
File "create_ffhq_lmdb.py", line 70, in <module>
main(args.split, args.ffhq_img_path, args.ffhq_lmdb_path)
File "create_ffhq_lmdb.py", line 46, in main
im = Image.open(img_path)
File "/usr/local/lib/python3.7/dist-packages/PIL/Image.py", line 2843, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/content/drive/MyDrive/Repositories/NVAE/dataset_nvae/ffhq/resized/55962.png'
Hi there and thanks for an amazing work! I was exploring the code and in the module distributions.py
found out that the logarithm of standard normal distribution (normalized_samples
in the code) is:
But why do we need a
Thanks in advance for the clarifications!
_norm_anneal=False, weight_decay_norm_init=10.0)
01/04 09:33:02 AM (Elapsed: 00:00:23) param size = 6.482514M
01/04 09:33:02 AM (Elapsed: 00:00:23) groups per scale: [10], total_groups: 10
01/04 09:33:02 AM (Elapsed: 00:00:23) epoch 0
/usr/bin/nvidia-modprobe: unrecognized option: "-s"
ERROR: Invalid commandline, please run /usr/bin/nvidia-modprobe --help
for
usage information.
/usr/bin/nvidia-modprobe: unrecognized option: "-s"
ERROR: Invalid commandline, please run /usr/bin/nvidia-modprobe --help
for
usage information.
In distributions.py
, the KL is computed as indicated in section 3.2 of the paper (residual normal distributions, Equation 2):
def kl(self, normal_dist):
term1 = (self.mu - normal_dist.mu) / normal_dist.sigma
term2 = self.sigma / normal_dist.sigma
return 0.5 * (term1 * term1 + term2 * term2) - 0.5 - torch.log(term2)
What I don't understand is, why you compute term2 = self.sigma / normal_dist.sigma
. Shouldn't it be:
term2 = self.sigma - normal_dist.sigma
?
Nice to meet you again ! Dr.@arash-vahdat ,
What does target = [0]
mean ? I can't figure out the place where this item used. And only find the returned img
data is fed into the NVAE model.
channel == 1
data to NVAE model ? I still can't find where the parameters denote the cin data channels. Or there don't need extra operations and Model can fit itself ? Still in confuse, beg your kindly help ! Sincerely !Hoping to get your help, Thank you again!
All the best,
Luke Huang
Hello! Can anyone explain me one thing: when counting mean3
for 3-rd channel (blue, I suppose), why don't we use samples[:, 2, :, :, :]
:
Lines 139 to 152 in 9fc1a28
Also, why do we need to update means with samples when counting log prob? For example, in Tacotron-2 code the are no updates of means.
Hi, I have no that powerful resources to train the models (2GPUs), could you provide pretrained models for evaluations purposes regarding the latent space? Thanks.
Hi all,
I've been testing the default code for training mnist and custom rgb_64x64 dataset.
In both cases, I can see the resulting training graphs, but the images in the tensorboard/images section are not showing -- they look like a 1x1 black pixel/empty.
The shape of the visualized image for mnist reconsturction is (1, int(sqrt(batch)), 2*int(sqrt(batch))) as expected, but is not shown. Saving the image with torchvision.utils.save_image()
saves the right size image.
What's the assumed tensorboard version?
tensorboard.version '1.15.0+nv'
torch.version '1.6.0a0+9907a3e'
torchvision.version '0.7.0a0'
Hi, in my experiment, I used Moving-MNIST dataset. But here are my problems during training that I couldn't find an answer:
I tried to play with a small network by using only num_latent_scale=1 and num_groups_per_scale=1. Then I realized there were no gradients generated for parameters including prior.ftr0 and an error was given to stop the training.
If I increase num_groups_per_scale from 1 to 2 or more, I still got Nan in some of the gradients in the first iteration, then they went away, but the training continues without errors.
I'm wondering if you could provide some hint or clue to why such behavior happens? Thank you in advance!
Hello,
I'm especially interested in the reconstruction of the images given as input to the model, and I would like to get and visualize the reconstructed images. To do so, is it the right way to use output_img = output.sample()
after logits, log_q, log_p, kl_all, _ = model(x)
and output = model.decoder_output(logits)
in the train and test functions ? Would you recommend to take several samples for the same image ?
Thank you for the work and the release,
Elsa
I am trying to implement the DDGAN paper. The authors ask the users to refer to this repository to for the dataset preparation.
The following command is suggested for downloading the dataset from openai's glow project:-
mkdir -p $DATA_DIR/celeba
cd $DATA_DIR/celeba
wget https://storage.googleapis.com/glow-demo/data/celeba-tfr.tar
tar -xvf celeba-tfr.tar
cd $CODE_DIR/scripts
python convert_tfrecord_to_lmdb.py --dataset=celeba --tfr_path=$DATA_DIR/celeba/celeba-tfr --lmdb_path=$DATA_DIR/celeba/celeba-lmdb --split=train
python convert_tfrecord_to_lmdb.py --dataset=celeba --tfr_path=$DATA_DIR/celeba/celeba-tfr --lmdb_path=$DATA_DIR/celeba/celeba-lmdb --split=validation
However when I implement the commands, I face the following errors:
`--2022-07-27 09:03:47-- https://storage.googleapis.com/glow-demo/data/celeba-tfr.tar
Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.4.128, 172.217.194.128, 142.251.10.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.4.128|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-07-27 09:03:48 ERROR 404: Not Found.
tar: celeba-tfr.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now`
Prafulla Dhariwal has addressed the some related errors here openai/glow/issues/1 for reference.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.