royorel/lifespan_age_transformation_synthesis

Lifespan Age Transformation Synthesis code

lifespan_age_transformation_synthesis's Issues

Making a good video with a not very powerfull graphic card

Dear Developers

I posse a GTX 950 M and i can only make 1-2 second long video with a "bad" quality. This is because i can set only a timestep of 0.15-2.

I ask if in the future it will be possible to make very good videos, like the gifs you posted, just making work the graphic card for a longer time. This is only to avoid the renting of a GPU cloud or buying a new graphic card, because of the actual crisis caused by COVID and mining that let a rising of prices.

Let us know!

Error in colab

Hello!
I wanted to test your code in the colab. Error when Downloading backbone Resnet Model parameters

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-10-5473477fe80b> in <module>()
----> 1 data_loader = CreateDataLoader(opt)
      2 dataset = data_loader.load_data()
      3 visualizer = Visualizer(opt)

5 frames
/content/Lifespan_Age_Transformation_Synthesis/data/data_loader.py in CreateDataLoader(opt)
     36     data_loader = AgingDataLoader()
     37     print(data_loader.name())
---> 38     data_loader.initialize(opt)
     39     return data_loader

/content/Lifespan_Age_Transformation_Synthesis/data/data_loader.py in initialize(self, opt)
     11     def initialize(self, opt):
     12         self.opt = opt
---> 13         self.dataset = CreateDataset(opt)
     14         self.dataloader = torch.utils.data.DataLoader(
     15             self.dataset,

/content/Lifespan_Age_Transformation_Synthesis/data/data_loader.py in CreateDataset(opt)
     29     dataset = MulticlassUnalignedDataset()
     30     print("dataset [%s] was created" % (dataset.name()))
---> 31     dataset.initialize(opt)
     32     return dataset
     33 

/content/Lifespan_Age_Transformation_Synthesis/data/multiclass_unaligned_dataset.py in initialize(self, opt)
     98 
     99         if (not self.opt.isTrain) and self.in_the_wild:
--> 100             self.preprocessor = preprocessInTheWildImage(out_size=opt.fineSize)
    101 
    102     def set_sample_mode(self, mode=False):

/content/Lifespan_Age_Transformation_Synthesis/util/preprocess_itw_im.py in __init__(self, out_size)
     48             print('Downloading backbone Resnet Model parameters')
     49             with requests.Session() as session:
---> 50                 download_file(session, resnet_file_spec)
     51 
     52             print('Done!')

/content/Lifespan_Age_Transformation_Synthesis/util/util.py in download_file(session, file_spec, chunk_size, num_attempts)
    102             # Validate.
    103             if 'file_size' in file_spec and data_size != file_spec['file_size']:
--> 104                 raise IOError('Incorrect file size', file_path)
    105             if 'file_md5' in file_spec and data_md5.hexdigest() != file_spec['file_md5']:
    106                 raise IOError('Incorrect file MD5', file_path)

OSError: [Errno Incorrect file size] deeplab_model/R-101-GN-WS.pth.tar

Running test on my own image

Hello, thank you for your good work. I am trying to test your code using my own image (single image) but it is always stuck at dataset [MulticlassUnalignedDataset] was created. Could you please help me find out the issue?
I ran the test code using python test.py instead through batch file because it has issue to access my library.
My system configurations are:

• Windows 10 64-bit.
• Nvidia RTX3090 CUDA 10.1.
• Pytorch 1.4.0 and torchvision 0.5.0.
• All libraries followed your requirements.txt.
  Thank you very much.

Any results on real life known people?

Are there any tests done on real life known people to estimate the relative accuracy of this work? For example giving images of a baby from 2000s and estimating his/her face now and comparing it to the person grown up now?

Thanks

Why does my model always generate video with only gray background when testing?

Hi, @royorel ,
I have prepared about 500 pictures for each age, so there are about 1500 in train0-2 subdir, 2000 in train3-6 subdir , etc. there are 10 levels in total. Start training:
python train.py --gpu_ids 0 --dataroot ./datasets/males --name males_model --batchSize 2 --verbose
After one epoch on Linux,loss_log.txt:
(epoch: 1, iters: 22440, time: 5.531) loss_G_Adv: 6.233 loss_G_Cycle: 2.508 loss_G_Rec: 2.522 loss_G_identity_reconst: 0.000 loss_G_age_reconst: 0.480 loss_D_real: 0.003 loss_D_fake: 0.002 loss_D_reg: 0.020
The three models generated are about the same size as your pre-trained models, and then tested on windows,use:
python test.py --name males_model --which_epoch latest --display_id 0 --traverse --interp_step 0.05 --image_path_file test/111.jpg --make_video --in_the_wild --verbose
However, the generated video have only gray background and no face. It's ok to change the path to your pre-trained models. The two model parameters which I print are the same,the model my program generate aboat 80M.
Why is this? look forward to your reply.

Error in colab

Hello!

I wanted to test your code in the colab. Everything worked fine, uploaded the file, but then there was some problem reading it:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-8c3f4407a6b4> in <module>()
----> 1 data = dataset.dataset.get_item_from_path(img_path)
      2 visuals = model.inference(data)
      3 
      4 os.makedirs('results', exist_ok=True)
      5 out_path = os.path.join('results', img_path[:-4] + '.mp4')

2 frames
/content/Lifespan_Age_Transformation_Synthesis/util/preprocess_itw_im.py in extract_face_landmarks(self, img)
     94 
     95         # Get the landmarks/parts for the face in box d.
---> 96         dlib_shape = self.predictor(img, dets[max_idx])
     97         landmarks = self.dlib_shape_to_landmarks(dlib_shape)
     98         return landmarks

TypeError: __getitem__(): incompatible function arguments. The following argument types are supported:
    1. (self: dlib.rectangles, s: slice) -> dlib.rectangles
    2. (self: dlib.rectangles, arg0: int) -> dlib.rectangle

Invoked with: rectangles[], -1

Result of Pre-trained model weight vs result of Train code

Result of Pre-trained model weight vs result of Train code

Question 1: The problem is that the test result with the provided weight is not the same as the test result after learning the provided train code. In particular, as a result of testing after learning with the train code, the face line in the image is not clear, the image quality is not clear overall, and the change of the face is also different.

Question 2: I made all the hyper-parameters the same and studied about 6 to 7 days with v100. The learning environment was set to gpu:4EA, epoch:400, batch-size: 6. How long did the learning time take for the author's experiment?

The first row below was tested based on the provided weight, and the second row was the training result of the provided train code.

Figure 1. Paper result

Figure 2. Train code result

Thank you for writing a good paper in the field of face age.

Continue Training Missing File

Hello, and thank you for sharing your work.

While attempting to use the continue training flag, I noticed two issues.

Firstly, when picking up from a previously run model, if the number of epochs specified is smaller than the amount already run, the model initializes and does not run without any explanation. After figuring out the number of epochs to continue training need to be added to the number previously run, it works fine, but this is very confusing and could be better explained in the documentation or the options.

Secondly, (and more critically) when that continue train flag is set, the model loader looks for two files named "<which_epoch>net<G/D>_tex.pth" in the model checkpoints folder. While a version without the '_tex' exists, neither of the _tex files do. Attempting to run as is produces the error:

 ./checkpoints/<model>/latest_net_G_tex.pth not exists yet!
Traceback (most recent call last):
  File "train.py", line 159, in <module>
    train(opt)
  File "train.py", line 42, in train
    model = create_model(opt)
  File "Lifespan_Age_Transformation_Synthesis/models/models.py", line 12, in create_model
    model.initialize(opt)
  File "Lifespan_Age_Transformation_Synthesis/models/LATS_model.py", line 94, in initialize
    self.load_network(self.netG, 'G_tex', opt.which_epoch, pretrained_path)
  File "Lifespan_Age_Transformation_Synthesis/models/base_model.py", line 62, in load_network
    raise('Generator must exist!')
TypeError: exceptions must derive from BaseException

This can be circumvented by renaming the existing network files to fit this naming scheme, but it concerns me none of the less. What is this _tex file? Should it have been generated or is it a naming scheme from a previous version?

My command for producing the error is below:

CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py --gpu_ids 0,1,2,3 --dataroot /data/ --name <model> --batchSize 16 --verbose --epochs 10 --save_epoch_freq 5 --continue_train --which_epoch latest

Got an error while training.

Original Traceback (most recent call last):
  File "/home/myname/anaconda3/envs/py37_v2/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/home/myname/anaconda3/envs/py37_v2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/myname/Lifespan_Age_Transformation_Synthesis/models/networks.py", line 799, in forward
    gan_out = self.gan_head(self.minibatch_stdev(features))
  File "/home/myname/Lifespan_Age_Transformation_Synthesis/models/networks.py", line 794, in minibatch_stdev
    out = torch.cat((input, mean_std), 1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 4 and 5 in dimension 2 at /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/THC/generic/THCTensorMath.cu:71

Training issues: Memory and training details

Hi! This is truly splendid work!

1. The first question is about the CUDA memory. Right now I have downloaded all the ffhq-aging dataset to my gdrive and try to train on colab (Tesla T4 was allocated to me this time, with 15GB memory). It turned out that only a batch size of 2 consumes 11GB memory on this graphic card and 3 will alert OOM. I was wondering is there any way to increase the batch size and speed up the training a little bit? E.g., modify some arguments in the options?

I noticed that it is said in the paper "Each model was trained with a batch size of 12 for 400 epochs on 4 GeForce RTX 2080 Ti GPUs." So I assume it is possible to train with at least a batch size of 3 on this card?

2. The second question is, during which epoch of training should I expect to see a human-face-like result? Maybe 100 epochs? As I am training very slowly, I am not sure when I will see some promising results :( There is nearly nothing in the first 2 epochs.

Sorry for my stupid question :) I am a new learner but really enjoy this kind of topic.

why the video my model produces changes from old to young and then becomes old?

hi, @royorel
I trained a model which cover all age stages,such as 0-2,3-6,7-9,10-14,15-19,20-29,30-39,40-49,50-69,70-120, and train about 160 epochs, then tested it like this:
python test.py --name males_model --which_epoch latest --display_id 0 --sort_order 0-2,3-6,7-9,10-14,15-19,20-29,30-39,40-49,50-69,70-120 --traverse --interp_step 0.05 --make_video --in_the_wild --verbose --image_path_file my.png
The video produced is that the face changes from old to young and then becomes old, not just from young to old. Why is this?
look forward to your reply！

Number of age bins

I noticed that you used 6 age bins to train. Would it be an issue if the bins are increased moderately, say to 7?
Or would you recommend training two models, covering 3 and 4 bins respectively ?

RuntimeError : CUDA out of memory

When I run ./run_scripts/in_the_wild.bat

I get the following error

RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 4.00 GiB total capacity; 960.24 MiB already allocated; 1.90 MiB free; 2.47 GiB reserved in total by PyTorch)

Please help to fix the error

Female model training cannot converge

Hello, when trying to reproduce the model, the male model starts to converge after training to 80 epochs, and gets better results when training to 400 epochs. At the same time, the female model was trained according to the same parameters, but there was no convergence trend until the 380th epoch. Does the female model have any special modifications in the training stage? Look forward to your reply!

Error in running the demo of google colab

Good morning dear developers, I have tried to run the demo that you provide in google colab but I have had an error about the CUDA memory, one more specification is that the error occurs in the following cell of the cell where the photo is uploaded:

data = dataset.dataset.get_item_from_path(img_path)
visuals = model.inference(data) <--------------- Here the error occurs

RuntimeError: CUDA out of memory. Tried to allocate 3.16 GiB (GPU 0; 11.17 GiB total capacity; 5.69 GiB already allocated; 758.81 MiB free; 9.91 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I have been following the instructions that you give to upload the photo, I do not think that is the problem but in the same way I leave the image that I have uploaded (this image belongs to the UTKFace dataset)

I hope you can help me, thank you

Training data arrangement

While training the model on some other face-aging dataset from scratch, how does the model expect the dataset arrangement to be?
More specifically, what should be the arrangement of folders in the "dataroot" folder?
It would be great if you could add that in the ReadMe, because it seems really difficult to figure that out through the data loader code.

Ps: Thanks for the great research. @royorel

Paste output into original image, original image age prediction

Hi Royorel

Really impressed by your work here! I am amazed by the accuracy of the outputs on some tests I did.

For my Python project, it would be even more amazing to be able to use the outputs combined with the original image, seeing the aging process on the photo itself basically. I tried digging around in the code, but I couldn't find a way to determine how the face is cut out of the image (basically I need to know "where" I should paste it over the original image). Is there any way you could help me out with this project?

As a side note, I was wondering if your algorithms can output the predicted age or some other way to determine the output image most fitting to the input. Is there something in the code that can give me an indication which pictures make the person older and which make the person younger?

Thanks a lot in advance for your helpful answers and keep up the astonishing work!

Solution for wrongly generated hair color/race

Hi there,
I tried using a picture of my grandpa (97 years old, hair all white) in your Colab notebook. It generated a video of my grandpa with white/blonde hair as a kid (an American kid), but he is Asian. Now is there a solution for this? Or is there any flag in the model regarding hair color/race/eye color that I don't know about?
I hope you guys can reply soon, thank you in advanced.

training model collapse

Hi, thanks for sharing your nice work and the dataset. I am playing around your code to know more about your idea. However, after training an epoch, the synthesized images of different classes are blank.

after about 1000 iterations

after 1 epoch

after 5 epoches

Training loss over time

I have not changed anything of your code. Do you have any idea about that? Thanks a lot.

Hard to reproduce pretrained model's results

Hi, royorel!
Thank you for sharing the excellent work! I want to reproduce the pretrained model's result, I encountered some issues. I use FFHQ 1024x1024 data, aligned by the method in 'download_ffhq_aging' script, and produced parsings image by your deeplab script. I train the model with female data use parsing to cover background, but cant converge a bit before 50epoch. loss_D_real and loss_D_fake decreasing to 0 fast. what do you think the reason? Hope for your suggestions.
the loss and training image:

why I stuck in the "create web directory ./checkpoint/males_model/web..."

I run the train.bat, but the code stuck in the "create web directory ./checkpoint/males_model/web..." like the picture show

Out of memory

Hello!

I've been trying to just run script on my machine and got out of memory error. Thing is, that i've tried to run it on 1080 ti with 11 gb free VRAM. What kind of graphics card needed to run this? Is there any solution to run script on current 1080ti?

Sorry, missed part in readme.

This is about just "GAN Training question" like "Lifespan GAN"

@royorel Hi,
I have a question about how to train a GAN.
It takes a lot of time to learn gan.

Do you test with an architecture suitable for 256x256 (high resolution) from the beginning to adjust the hyper-parameter or architecture?

Or

Do you experiment with a small image such as 64x64 (low resolution), adjust the hyper-parameter first, and then train by reducing the number of hyper-parameter cases by creating an architecture that increases the resolution?

Thanks for reading

How i can get high quality images/video output? it's blocked to 256 currently

i have tried modify loadsize and finesize in base_options.py to 600 from 256 but nothing happen it still blocked default 256 only

CUDA out of memory when running the demo.

Hi, thanks for sharing your nice work and the dataset. I am playing around your code to know more about your idea. But when running the demo locally or on Google colab, it randomly prompts CUDA out of memory like this:

I have not changed anything of your code. Do you have any idea about that? Thanks a lot.

RuntimeError: CUDA out of memory

Tried with three different images, always face this issue in colab. Resizing to 100x100 didn't help:
RuntimeError: CUDA out of memory. Tried to allocate 3.21 GiB (GPU 0; 11.17 GiB total capacity; 4.47 GiB already allocated; 2.45 GiB free; 8.26 GiB reserved in total by PyTorch)

Resizing to 64x64 yields to no face detecting.

11GB Vram isn't enough?

i want to generate 6 images only not the video

hoe to generate six images only not the gif video and how to download the model as pkl file to use it

why ValueError: not enough values to unpack (expected 10, got 7)

hi， I run the code on the linux, there's some issues