mutalyzer/retriever

retriever's Issues

Biopython version 1.80

Some tests fail.

mutalyzer_retriever - Email address is not specified, HTTP Error 404: Not Found and mutalyzer_retriever.retriever.NoReferenceError

Dear,

Since 28/03/2024, mutalyzer_retriever command cannot retrieve references.

I tested v0.2.4, v0.2.7, v0.3.0 and v0.3.3 of Mutalyzer on our CentOS and Ubuntu servers.

Here the two erros I have:
v2.0.4

/home/localadmin/.local/lib/python3.8/site-packages/Bio/Entrez/__init__.py:726: UserWarning:
            Email address is not specified.

            To make use of NCBI's E-utilities, NCBI requires you to specify your
            email address with each request.  As an example, if your email address
            is [email protected], you can specify it as follows:
               from Bio import Entrez
               Entrez.email = '[email protected]'
            In case of excessive usage of the E-utilities, NCBI will attempt to contact
            a user at the email address provided before blocking access to the
            E-utilities.
  warnings.warn(
Traceback (most recent call last):
  File "/home/localadmin/.local/bin/mutalyzer_retriever", line 12, in <module>
    sys.exit(main())
  File "/home/localadmin/.local/lib/python3.8/site-packages/mutalyzer_retriever/cli.py", line 220, in main
    _endpoint(args)(args)
  File "/home/localadmin/.local/lib/python3.8/site-packages/mutalyzer_retriever/cli.py", line 184, in _retrieve_raw
    output = retrieve_raw(
  File "/home/localadmin/.local/lib/python3.8/site-packages/mutalyzer_retriever/retriever.py", line 124, in retrieve_raw
    reference_content, reference_type, reference_source = _fetch_unknown_source(
  File "/home/localadmin/.local/lib/python3.8/site-packages/mutalyzer_retriever/retriever.py", line 99, in _fetch_unknown_source
    _raise_error(status)
  File "/home/localadmin/.local/lib/python3.8/site-packages/mutalyzer_retriever/retriever.py", line 55, in _raise_error
    raise NoReferenceError(status, uncertain_sources)
mutalyzer_retriever.retriever.NoReferenceError:

Uncertain sources: ncbi

lrg:
 NameError: (HTTPError)(HTTP Error 404: Not Found)
ncbi:
 ConnectionError:
ensembl:
 NameError:

>=v0.2.7

Traceback (most recent call last):
  File "/usr/local/bin/mutalyzer_retriever", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/mutalyzer_retriever/cli.py", line 124, in main
    timeout=args.timeout,
  File "/usr/local/lib/python3.6/site-packages/mutalyzer_retriever/retriever.py", line 114, in retrieve_raw
    reference_id, reference_type, size_off, timeout
  File "/usr/local/lib/python3.6/site-packages/mutalyzer_retriever/retriever.py", line 85, in _fetch_unknown_source
    _raise_error(status)
  File "/usr/local/lib/python3.6/site-packages/mutalyzer_retriever/retriever.py", line 43, in _raise_error
    raise NoReferenceError(status)
mutalyzer_retriever.retriever.NoReferenceError:
lrg: NameError.
ncbi: ConnectionError.
ensembl: NameError.

Is there something updated in Ensembl and NCBI? Do you know how to fix it?

These commands does not work:

mutalyzer_retriever --id "NM_020937.3" --parse
mutalyzer_retriever --id 'NM_020937.3' --parse
mutalyzer_retriever --id NM_020937.3 --parse
mutalyzer_retriever --id AC254562.1 --parse
mutalyzer_retriever --id NM_001320134.1 --parse
mutalyzer_retriever --id NM_020937.3 --related --indent 2
mutalyzer_retriever --id NM_020937.3 -s ensembl
mutalyzer_retriever --id NM_020937.3 -s ncbi

These commands work:

mutalyzer_retriever  --id LRG_303 --parse
mutalyzer_retriever --id LRG_303 --related --indent 2
mutalyzer_retriever --id LRG_303 -s lrg

I also tried to define an adress email in mutalyzer_retriever and /home/localadmin/.local/lib/python3.8/site-packages/Bio/Entrez/init.py:

from Bio import Entrez
Entrez.email = '[email protected]'
email = '[email protected]'

Best,
Vimel

Features not extracted

Exon and transcript features are not extracted for XR_948219.2. Seems to be due to the gff parsing library.

Support for older Ensembl versions

Instead of the currently used APIs (GRCh38 and GRCh37) we could switch to Ensembl Tark.

Remove biopython dependency?

Check if we can drop the biopython dependency.

Missing support for mitochondrial genes.

Mitochondrial genes have no transcripts in the classical sense, so no NM_ identifiers are present in the annotation. We should therefore allow for other ways to select a CDS, the name of the gene seems to be the most practical candidate.

Please see the following example: NC_012920.1(ND4):c.10del

Retrieving assemblies issue

The retriever fails to retrieve assemblies annotations since the FTP location includes new directories with a different structure.

Configuration

NCBI API key and email are not properly set.

Own GFF parser

Given that the current GFF parser is not really actively maintained, it may be a good idea to consider developing our own parser.