ReLERNN_TRAIN_POOL is slow

[kr-colab]

ReLERNN_TRAIN_POOL is slow

[klucek]

Hi, I have a poolseq data set that I would like to analyze. However, even with the training data set I encounter the same error message

Epoch 1/2

^M 1/100 [..............................] - ETA: 15:44 - loss: 1.1623^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^$ 2020-05-06 13:58:50.410794: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
Epoch 2/2
^M 1/100 [..............................] - ETA: 9:11 - loss: 1.1022^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H$ 2020-05-06 14:10:56.902371: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
findfont: Font family ['serif'] not found. Falling back to DejaVu Sans.

The program nevertheless continues, but can these outputs be taken serious given the errors above?

[jradrion]

Hi @klucek,

I haven't had found that the tensorflow error Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled changes the results, but I'm not sure about the ^H^Hs.

Can you check to see if a file named ./networks/testResults.pdf was created? If so, then you should be able to see a plot of the training history on the bottom, and the predictions on the test set on the top. Provided those look normal, and the model and weights were created (./networks/model.json & ./networks/weights.h5), you should be able to ignore the above errors.

I'm hoping that these error messages will be resolved when I'm able to move to an updated tensorflow version.

[jradrion]

I added optional muti-threading to model.fit for training on pool-seq data. Multi-threading can be disabled by simply using the default option for --nCPU in ReLERNN_TRAIN and ReLERNN_TRAIN_POOL. Everything seems to be working using tensorflow 2.2, which is now the supported version via the README. I'm going to go ahead and close this issue now.