nbisweden/libpredweb

Library for prediction web-server

libpredweb

This is the library for the protein prediction servers used by the following repos

• https://github.com/NBISweden/predictprotein-webserver-topcons2
• https://github.com/NBISweden/predictprotein-webserver-proq3
• https://github.com/NBISweden/predictprotein-webserver-scampi2
• https://github.com/NBISweden/predictprotein-webserver-boctopus2
• https://github.com/NBISweden/predictprotein-webserver-pconsc3
• https://github.com/NBISweden/predictprotein-webserver-prodres
• https://github.com/NBISweden/predictprotein-webserver-subcons
• https://github.com/NBISweden/predictprotein-webserver-common-backend

How to install

Use the following command to install the package with pip

pip install git+https://github.com/nanjiangshu/libpredweb.git@main

Usage

Add the following lines to your Python script in order to use the library

from libpredweb import myfunc
from libpredweb import webserver_common as webcom
from libpredweb import qd_fe_common as qdcom

libpredweb's Issues

Investigate the possibilities to use Javascript to create server statistics plot

For the server usage statistics plots at e.g. https://topcons.net/pred/serverstatus/, javascript might be a good option for interactive plots.

Recreate VMs on the new RECAS-BARI site

The old RECAS-BARI cloud is down and it has been moved to a new site https://horizon.recas.ba.infn.it/horizon.
VMs should be recreated on the new cloud site, this includes

1. Rshiny
2. prediction server backend node (2x)
3. dev server

webcom.WriteDumpedTextResultFile not working when name_software is with prefix

When name_software has prefix docker_ or singularity_ WriteDumpedTextResultFile is not working as expected

Investigate why subcons failes to predict for some queries

The subcons web-server is failed to predict some of the queries,

such as

>t2
IEVYHSYFWPLEWTIPSRDNNKCYAKIICNTKDNERVVGFHVLGPNAGEVTQGFAAALKCMAQRLLLRRFLASVISRKPSQGQWPPLTSRALQTPQCSPGGLTVTPNPARTIYTTRISLTTFNIQDGPDFQDRVVNSETPVVVDFHAQWCGPCKILGPRLEKMVAKQHGKVVMAKVDIDDHTDLAIEYEAGSLSPRLECSGTITAH

>lcl|ORF34
MEIEGPLPNHHGNIPKNVPFLIGQKARYLFLSVFSMAVIYFTRSPLMGLTHFSAYILITGCIFSFSTLDDTLSSQPTVQCFYGHAHSSTILVPQYFSLQSQQSIPLLPPGMATPCLATNIAKLDTTSTLLTGNRCCSHLCHCCRSKLSLPTTSMSAFFYHQFSVHCSVYL

However, it works with the default example

>sp|P04201|MAS_HUMAN
MDGSNVTSFVVEEPTNISTGRNASVGNAHRQIPIVHWVIMSISPVGFVENGILLWFLCFRMRRNPFTVYITHLSIADISLLFCIFILSIDYALDYELSSGHYYTIVTLSVTFLFGYNTGLYLLTAISVERCLSVLYPIWYRCHRPKYQSALVCALLWALSCLVTTMEYVMCIDREEESHSRNDCRAVIIFIAILSFLVFTPLMLVSSTILVVKIRKNTWASHSSKLYIVIMVTIIIFLIFAMPMRLLYLLYYEYWSTFGNLHHISLLFSTINSSANPFIYFFVGSSKKKRFKESLKVVLTRAFKDEMQPRRQKDNCNTVTVETVV

Figure out the problem and try to fix it.

Renew Grid certificate

The grid certificate is going to expire and try to renew it through

https://cert-manager.com/customer/sunet/idp/clientgeant

Investigate why SSL certificate is not updated by the cron job

As a developer, I want SSL certificates of web-servers are automatically updated by crontab jobs on both pcons1 and pcons2.

Hosts:
pcons1.scilifelab.se
pcons2.scilifelab.se

Create backend node for proq3 and scampi2 other than pcons3

Currently, pcons3 is the only backend node for proq3 and scampi2, create at least one more node for the these prediction methods.

pcons3 is not accessible

pcons3 server is not accessible. The IT has made a hard reboot but it is still not accessible.

duplicated processing for archived results

Retrieving results from archived results is sometimes run twice, the consequence is that duplicated records are found in the file finished_seqs.txt.

DoD: The archived results are processed only once

Fix undefined name_server in GetNameServerFromNameSoftware for scampi2

For scampi2, name_server is not set

Hints: the name_software for scampi2 is either scampi2-single or scampi2-msa

use new pdb2seq function

The old pdb2seq() function based on biopython PDBparser has a bug when the PDB file has atom record containing e.g. HD11, that is, the 13th column is not empty and with letter 'H'.

We need to write a new function that will solve this problem

DOD: the new pdb2seq() function parse the PDB files correctly for even the aforementioned cases.

Bug: the variable "status" is of conflict

In the method GetResult(), the value of variable status has been changed in the inner block by mistake.

Investigate the possibilities to use Javascript to create server statistics plot

For the server usage statistics plots at e.g. https://topcons.net/pred/serverstatus/, javascript might be a good option for interactive plots.

Show all sequences in the table on the result page

As a user, I want to see all sequences of my job listed in the table on the result page (see the attached screenshot) as well as an example from TOPCONS https://topcons.net/pred/result/rst_b9l2iwo5/ (this link might not be available after a while since the finished result will be deleted after certain amount of time).

One solution might be use server side processing.

Use the new function InitCounterSubmitJobDict in all qd_fe.py

The function InitCounterSubmitJobDict has been created in the library. This has to be used for qd_fe.py of all other web-servers, including

• proq3
• topcons2
• boctopus2
• scampi2
• pathopred.bioinfo.se
• pconsc3
• prodres
• subcons

INFN-BARI site is down

INFN-BARI site is down, contact the resource provider to solve the problem

Fix unexpected behavior of MAX_TIME_IN_REMOTE_QUEUE

For jobs submitted to the remote queue that are not running, it is designed that when

queue_time > MAX_TIME_IN_REMOTE_QUEUE

The job will be deleted. This is failed sometimes. This needs to be fixed.

Lock file not always cleaned for python pipelines

Description
Some pipelines may be blocked from running since the lock files are not cleaned after unexpected exit. This should be fixed so that the lock files are always cleaned at exit, even when it's an abnormal exit.

Applied Python scripts

• clean_cached_result.py
• job_final_process.py
• run_server_statistics.py

Fix the pipeline on the computational node for PconsC3

The pipeline on the computational node for pconsc3 is currently broken.

The problem is that after VM upgrading and pipeline recompilation, the perl for the packages psipred and pfamscan become incompatible.

Login to the VM

ssh [email protected]

Re-build VM for the computational node on RECAS-BARI

As a developer, I want to the computational node on the RECAS-BARI work as expected. The previous setting with shared NFS storage is good for scalability, but the VM crashes frequently because the intensive I/O on the NFS storage. Therefore, the VMs on those nodes needs to be rebuild without shared storage.

Add support for SCAMPI web server

Add support for SCAMPI web-server in the library to reduce duplicated code.

Daemon exits abnormally when the archived result zip file is bad (empty)

The daemon script qd_fe.py exits abnormally when the archived result zip file is bad (empty)

DoD: the script should handle is situation and keep running.

Add support for scampi2 for CreateRunJoblog

Add support for SCAMPI2 in the function CreateRunJoblog

Bad plots for server statistics

The plots for server statistics (https://topcons.net/pred/serverstatus/) are out of shape for TOPCONS and probably also for other web-servers.

Notes
These plots are created by Gnuplots.

Fix vulnerabilities with Django

As a developer, I want Django to be updated so that the vulnerabilities are resolved.

Related links:

https://github.com/NBISweden/predictprotein-webserver-boctopus2/security/dependabot/requirements.txt/django/open
https://github.com/NBISweden/predictprotein-webserver-scampi2/security/dependabot/requirements.txt/django/open

It is probably valid for the other web-servers as well.

For SCAMPI and BOCTOPUS2, please note that the current code base is not compatible with the latest Django, i.e. 2.2.24.

• pconsc3
• scampi
• topcons
• boctopus
• proq3
• subcons
• prodres
• common_backend

add function urlretrieve with timeout

the urllib version of urlretrieve will hang when the network is bad. We need to add a function for downloading file from url with timeout.

DOD: add a function urlretrieve in the module myfunc with timeout option.

port security for the backend server

Use port other than 80 for the backend servers

original request from the EGI provider

the following alert has been received from GARR about some issues on your instance on ReCaS-Bari Cloud with IP 212.189.205.55 (TENANT: EGI_bils). We kindly ask you to solve urgently since the deadline is on November 18th 2022. The issues are reported below:
-----------

User: [e-mail]_egiID
Tenant: EGI_bils

--- Inizio REPORT ---

Sede: ba
Ip: 212.189.205.55
SSLv2: 
SSLv3: 
TLS1: 
TLS1.1: 
TLS1.2: 
TLS1.3: 

--- Fine REPORT -----

TO KEEP YOUR CONFIGURATION SAFE: 
   
 1) if a service is ACTIVE on port 80 THERE MUST BE a redirection (automatically or not)
to port 443 
   
 2) if a service is ACTIVE on port 443 MUST
    * ACCEPT only the following protocols:
      - TLS 1.2 
      - TLS 1.3 
    * REJECT the following protocols
      - SSLv2  
      - SSLv3 
      - TLS1 
      - TLS1.1 
 
The following tools are available:  
  * to safely configure the server : 
    https://ssl-config.mozilla.org/ 
  * to check the solution adopted: 
    https://github.com/drwetter/testssl.sh/tree/3.0 

We kindly ask you to give as a feedback about the actions undertaken as soon as posslible.
Regards

ReCaS-Bari Support Team

Run job completion procedure in a separate process for TOPCONS

When a job contains many (e.g. >1000) sequences, the procedure to create the dumped result and making a zip file may take quite some time and thus blocking other jobs. It will be better to run the procedure in a separate process, in a queuing system.

Related code:
https://github.com/NBISweden/libpredweb/blob/master/libpredweb/qd_fe_common.py#L1376-L1437
https://github.com/NBISweden/predictprotein-webserver-topcons2/blob/master/proj/pred/app/qd_fe.py#L1164-L1234

change default branch from master to main

As a developer, I want to change the default branch from master to main

fix errors in the readme

This errors in the readme

Backup web-server usage data files

As a developer, I want a proper way to backup the usage data files for prediction web-servers.
These files were previously backed up on the computer shu.scilifelab.se but it has been shut down now.

Fix the pipeline on the computational node for PconsC3

The pipeline on the computational node for pconsc3 is currently broken.

The problem is that after VM upgrading and pipeline recompilation, the perl for the packages psipred and pfamscan become incompatible.

Login to the VM

ssh [email protected]

The daemon script trapped in bad loop when the first node is not accessible

The daemon is not forwarding when the first backend node is not accessible.

The problem should be related to here

https://github.com/NBISweden/libpredweb/blob/main/libpredweb/qd_fe_common.py#L579-L584

Fix submission statistics in server status page

The submission statistics shows only submitted jobs via the web interface. It should include jobs from both the web interface and the API.

Integrate server uptime monitoring with Slack

As a developer, I want the uptime monitoring of web-servers integrate to Slack so that every team member can see the status these web-servers.

List of web-servers to be monitored:

https://topcons.net
https://proq3.bioinfo.se
https://subcons.bioinfo.se
https://scampi.bioinfo.se
https://b2.topcons.net
https://prodres.bioinfo.se
https://pconsc3.bioinfo.se

Code refactoring: reuse code from libpredweb for TOPCONS

As a developer, I want to reuse the code as much as possible from the library libpredweb without affect current functions.

Checklist

• topcons

Scampi2 and proq3 can be solved in a separate issue.