Sir
I tried to run it on 8GB GPU(Active 7 GB) V100D, with 64 GB RAM,
But i got unknow error: during model.fit
Loading Convolution Algorithm is failed due to memory constraints even in the case of batch_size=1.
kindly suggest me in this regard
UnknownError Traceback (most recent call last)
in
----> 1 model.fit(data, [labels, data], batch_size=1, epochs=1)
~.conda\envs\Gandhiji\lib\site-packages\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
1176 steps_per_epoch=steps_per_epoch,
1177 validation_steps=validation_steps,
-> 1178 validation_freq=validation_freq)
1179
1180 def evaluate(self,
~.conda\envs\Gandhiji\lib\site-packages\keras\engine\training_arrays.py in fit_loop(model, fit_function, fit_inputs, out_labels, batch_size, epochs, verbose, callbacks, val_function, val_inputs, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps, validation_freq)
202 ins_batch[i] = ins_batch[i].toarray()
203
--> 204 outs = fit_function(ins_batch)
205 outs = to_list(outs)
206 for l, o in zip(out_labels, outs):
~.conda\envs\Gandhiji\lib\site-packages\keras\backend\tensorflow_backend.py in call(self, inputs)
2977 return self._legacy_call(inputs)
2978
-> 2979 return self._call(inputs)
2980 else:
2981 if py_any(is_tensor(x) for x in inputs):
~.conda\envs\Gandhiji\lib\site-packages\keras\backend\tensorflow_backend.py in _call(self, inputs)
2935 fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)
2936 else:
-> 2937 fetched = self._callable_fn(*array_vals)
2938 return fetched[:len(self.outputs)]
2939
~.conda\envs\Gandhiji\lib\site-packages\tensorflow\python\client\session.py in call(self, *args, **kwargs)
1456 ret = tf_session.TF_SessionRunCallable(self._session._session,
1457 self._handle, args,
-> 1458 run_metadata_ptr)
1459 if run_metadata:
1460 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node Input_x1/convolution}}]]
[[loss/Dec_VAE_Output_loss/Mean_3/_2353]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node Input_x1/convolution}}]]
0 successful operations.
0 derived errors ignored.