mamba installation problem

[gaworj]

mamba installation problem

[rpetit3]

I just tried and it worked. I am however using Miniconda. I've attached an environment export to try if you want.

Otherwise, does anything change if you also pin tensorflow-gpu?

mamba create -n dragonflye -c conda-forge -c bioconda 'dragonflye=1.0.13' 'tensorflow-gpu>=2.*'

dragonflye-env.txt

[gaworj]

I have followed your suggestion:

mamba create -n dragonflye -c conda-forge -c bioconda 'dragonflye=1.0.13' 'tensorflow-gpu>=2.*'

              __    __    __    __
             /  \  /  \  /  \  /  \
            /    \/    \/    \/    \

███████████████/ /██/ /██/ /██/ /████████████████████████
/ / \ / \ / \ / \ ____
/ / _/ _/ _/ \ o _,
/ / ___/ `
|/
███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗
████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
██╔████╔██║███████║██╔████╔██║██████╔╝███████║
██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║
╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝

    mamba (0.22.1) supported by @QuantStack

    GitHub:  https://github.com/mamba-org/mamba
    Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████

Looking for: ['dragonflye=1.0.13', "tensorflow-gpu[version='>=2.*']"]

r/linux-64 No change
bioconda/noarch 3.7MB @ 3.2MB/s 1.2s
pkgs/main/linux-64 No change
conda-forge/noarch 9.5MB @ 4.8MB/s 2.1s
pkgs/r/noarch No change
r/noarch No change
pkgs/r/linux-64 No change
pkgs/main/noarch No change
cruizperez/linux-64 No change
bioconda/linux-64 4.4MB @ 1.2MB/s 3.6s
conda-forge/linux-64 25.8MB @ 4.1MB/s 6.8s
cruizperez/noarch No change
Encountered problems while solving:

• package tensorflow-gpu-2.0.0-h0d30ee6_0 requires tensorflow 2.0.0, but none of the providers can be installed

Second option:

conda create -n dragonflye --file dragonflye.txt

CondaValueError: could not parse 'name: dragonflye' in: dragonflye.txt

..?

[rpetit3]

Try this: conda env create -f dragonflye.txt

[gaworj]

File extension format was wrong:

conda env create -f dragonflye.yml

Installation in progress...

[rpetit3]

Apologies, should have mention filename change was needed. GitHub doesn't all yaml files to be uploaded

[gaworj]

OK, so now I have correct dragonflye version but encopuntered medaka issue:

[polishing - medaka (1 of 2)] Running medaka consensus
[polishing - medaka (1 of 2)] [18:46:50 - Predict] Reducing threads to 2, anymore is a waste.
[polishing - medaka (1 of 2)] [18:46:50 - Predict] Setting tensorflow inter/intra-op threads to 2/1.
[polishing - medaka (1 of 2)] [18:46:50 - Predict] Processing region(s): contig_1:0-2039123 contig_2:0-19600 contig_3:0-18712
[polishing - medaka (1 of 2)] [18:46:50 - Predict] Using model: /home/jang/anaconda3/envs/mamba/envs/dragonflye/lib/python3.8/site-packages/medaka/data/r941_min_sup_g507_model.tar.gz.
[polishing - medaka (1 of 2)] [18:46:50 - Predict] Found a GPU.
[polishing - medaka (1 of 2)] [18:46:50 - Predict] If cuDNN errors are observed, try setting the environment variable TF_FORCE_GPU_ALLOW_GROWTH=true. To explicitely disable use of cuDNN use the commandline option `--disable_cudnn. If OOM (out of memory) errors are found please reduce batch size.
[polishing - medaka (1 of 2)] [18:46:50 - Predict] Processing 5 long region(s) with batching.
[polishing - medaka (1 of 2)] [18:46:50 - ModelLoad] GPU available: building model with cudnn optimization
[polishing - medaka (1 of 2)] [18:46:50 - MdlStrTF] ModelStoreTF exception <class 'tensorflow.python.framework.errors_impl.InternalError'>
[polishing - medaka (1 of 2)] Traceback (most recent call last):
[polishing - medaka (1 of 2)] File "/home/jang/anaconda3/envs/mamba/envs/dragonflye/bin/medaka", line 11, in
[polishing - medaka (1 of 2)] sys.exit(main())
[polishing - medaka (1 of 2)] File "/home/jang/anaconda3/envs/mamba/envs/dragonflye/lib/python3.8/site-packages/medaka/medaka.py", line 720, in main
[polishing - medaka (1 of 2)] args.func(args)
[polishing - medaka (1 of 2)] File "/home/jang/anaconda3/envs/mamba/envs/dragonflye/lib/python3.8/site-packages/medaka/prediction.py", line 160, in predict
[polishing - medaka (1 of 2)] model = model_store.load_model(time_steps=args.chunk_len)
[polishing - medaka (1 of 2)] File "/home/jang/anaconda3/envs/mamba/envs/dragonflye/lib/python3.8/site-packages/medaka/datastore.py", line 159, in load_model
[polishing - medaka (1 of 2)] self.model = model_partial_function(time_steps=time_steps)
[polishing - medaka (1 of 2)] File "/home/jang/anaconda3/envs/mamba/envs/dragonflye/lib/python3.8/site-packages/medaka/models.py", line 144, in build_model
[polishing - medaka (1 of 2)] gru = GRU(
[polishing - medaka (1 of 2)] File "/home/jang/anaconda3/envs/mamba/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent_v2.py", line 379, in init
[polishing - medaka (1 of 2)] if context.num_gpus() > 0:
[polishing - medaka (1 of 2)] File "/home/jang/anaconda3/envs/mamba/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 2046, in num_gpus
[polishing - medaka (1 of 2)] return context().num_gpus()
[polishing - medaka (1 of 2)] File "/home/jang/anaconda3/envs/mamba/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 1047, in num_gpus
[polishing - medaka (1 of 2)] self.ensure_initialized()
[polishing - medaka (1 of 2)] File "/home/jang/anaconda3/envs/mamba/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 515, in ensure_initialized
[polishing - medaka (1 of 2)] context_handle = pywrap_tfe.TFE_NewContext(opts)
[polishing - medaka (1 of 2)] tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
[polishing - medaka (1 of 2)] Failed to run medaka consensus.
[dragonflye] Error running command: medaka_consensus -i READS.fq.gz -d flye/polish/racon/2/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 12 2>&1 | sed 's/^/[polishing - medaka (1 of 2)] /' | tee -a dragonflye.log

Is it possible to force dragonflye to use medaka on cpus?

[rpetit3]

Full disclaimer, I have no GPU to test things on. But, I think you might need to update your cuda drivers on the system.

Otherwise, you can try adding --medaka_opts '--disable_cudnn'

[gaworj]

Added --medaka_opts '--disable_cudnn'

[dragonflye] Polishing with Medaka (2 rounds)
[dragonflye] Running: medaka_consensus -i READS.fq.gz -d flye/polish/racon/2/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 12 --disable_cudnn 2>&1 | sed 's/^/[polishing - medaka (1 of 2)] /' | tee -a dragonflye.log
[polishing - medaka (1 of 2)] Invalid option: --.
[dragonflye] Error running command: medaka_consensus -i READS.fq.gz -d flye/polish/racon/2/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 12 --disable_cudnn 2>&1 | sed 's/^/[polishing - medaka (1 of 2)] /' | tee -a dragonflye.log

There is no option --disable_cudnn in medaka_consensus help.

[rpetit3]

Looking at this: https://datascience.stackexchange.com/questions/58845/how-to-disable-gpu-with-tensorflow

You might need to do this:

export CUDA_VISIBLE_DEVICES="-1"

OR 

export CUDA_VISIBLE_DEVICES=''

[gaworj]

After setting export CUDA_VISIBLE_DEVICES='' medaka finished successfully. Thanks a lot for your help!

Can you also add --prefix option in dragonflye to set the name of the final consensus file?

[rpetit3]

hi @gaworj

--prefix is available in v1.0.14. I'm going to go ahead and close this issue.

Cheers,
Robert