gretel not finding GPU

[gretelai]

gretel not finding GPU

[johntmyers]

If you import TF yourself whats your output from:

physical_devices = tf.config.list_physical_devices('GPU')
print("Num GPUs:", len(physical_devices))

and also

tf.test.gpu_device_name()

[tmontana]

Here is the output:

physical_devices = tf.config.list_physical_devices('GPU')
print("Num GPUs:", len(physical_devices))

Num GPUs: 1

tf.test.gpu_device_name()

'/device:GPU:0'

If you import TF yourself whats your output from:

physical_devices = tf.config.list_physical_devices('GPU')
print("Num GPUs:", len(physical_devices))

Num GPUs: 1

and also

tf.test.gpu_device_name()

'/device:GPU:0'

thanks

[johntmyers]

Thanks, did you run this in the example notebook that you were trying to run or somewhere else?

Also, that warning doesn't change anything under the hood, it just prints the warning in the event the tf.test.gpu_device_name() does not return the expected value. The model should still start to train regardless. If you let the model train, how long does each epoch take to run?

[tmontana]

Thanks, did you run this in the example notebook that you were trying to run or somewhere else?

Also, that warning doesn't change anything under the hood, it just prints the warning in the event the tf.test.gpu_device_name() does not return the expected value. The model should still start to train regardless. If you let the model train, how long does each epoch take to run?

Ok my bad - for some reason the problem is linked to my conda environment. I had run your commands in the Python3 environment. If I run them in the Gretel environment (created as above) I get 'Num GPUs: 0'.

Would you know why TF is not seeing the GPU?

Thanks

[johntmyers]

I'm not sure, I've personally only used a virtualenv to run things. @zredlined have you run into this with conda or had to do any other steps to allow the GPU to be accessed in an environment like that?

[zredlined]

Tensorflow versions are coded to different CUDA library versions. Conda's pretty good at handling this- so we usually recommend setting up tensorflow through Conda.

Here's a link to the pinned TF->CUDA versions.

Try installing dependencies with Conda.

conda install tensorflow=2.3 cudatoolkit=10.1
pip install gretel-synthetics

[tmontana]

still doesn't work. Anyway it's a tensorflow issue apparently (and not Gretel) as the GPU is not recognized after I install just tensorflow. I'll investigate it further. Thank you

[zredlined]

Looks like there might be some dependency mismatches in the latest Conda scripts. Here are instructions for building a tensorflow==2.4 virtual environment with Conda, gretel-synthetics, and GPU. I tested on AWS running a debian image. Hope this helps!

First, setup your Conda environment
conda create --name tf --python=3.8
conda activate tf

Copy this Gist using the TensorFlow team's latest instructions into a shell script named setup_deps.sh: https://gist.github.com/zredlined/dc7a0cb5ca72d58a8cc29ce6eee441cf

Next, run the setup script
sh setup_deps.sh

[tmontana]

sh setup_deps.sh

thanks @zredlined - that worked. I've been on it for the past 3 hours and was going to give up. I tested on AzureML and working fine. Cheers