Prosit prediction is not deterministic on RTX3070

[courcelm]

Prosit prediction is not deterministic on RTX3070

[LLautenbacher]

Hi, no, that is not expected. If you can, provide the Task IDs for your job, I can take a look.

[courcelm]

Hello, thanks for your quick answer. I don't see the problem on your website but with my local deployment.

I started the Prosit server with this command:

docker run --gpus all -it
-v /mnt/projects/MC2021-09-20/prosit/prosit_non_tryptic_fragmentation/hla_hcd:/root/model_spectra/
-v /mnt/projects/MC2021-09-20/prosit/model_irt_prediction:/root/model_irt/
-e CUDA_VISIBLE_DEVICES=0
-p 5000:5000
prosit python3 -m prosit.server

I have this input file for prediction:

modified_sequence,collision_energy,precursor_charge
VDASIFKAW,30,2

Then I run the prediction with this command:

curl --form peptides=@peptidelist_1.csv http://127.0.0.1:5000/predict/generic

Each time I run it, I get a different result.

[courcelm]

I found a previous report about this kind of issue with TF and Keras here:
keras-team/keras#12800

Here is how I fixed the issue for Prosit:

New Dockerfile:

FROM nvcr.io/nvidia/tensorflow:21.09-tf1-py3


RUN apt-get update -qq && \
    apt-get install -y \
            libhdf5-serial-dev \
    && rm -rf /var/lib/apt/lists/*


RUN pip install keras==2.2.4 h5py tables flask pyteomics lxml pandas

ENV KERAS_BACKEND=tensorflow
ENV TF_CPP_MIN_LOG_LEVEL=3
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

ENV TF_DETERMINISTIC_OPS=1

ADD prosit/ /root/prosit
RUN cd /root/
WORKDIR /root/

I can now run Prosit on my NVIDIA Geforce RTX3070.