biocyberman/bcbio-nextgen

Best-practice pipelines for fully automated high throughput sequencing analysis

bcbio-nextgen's Issues

how to dynamically locate ericscript resources based on genome build?

@chapmanb Brad and @tetianakh Tetiana

Need your help to pass test cases of ericscript. The part of dynamic locating of resource I don't understand very much yet.

I already download ericscript data and have them saved under:
/<bcbioinstalldir>/genomes/Hsapiens/hg38/rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens
and
/<bcbioinstalldir>/genomes/Hsapiens/hg19/rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens

However, the pytest failed, and I think because of incorrect data resources setup, and self._db_location is None.

I have this in the end of /<bcbio_installationdir>/galaxy/bcbio_system.yaml

 ericscipt:
    db: /<bcbioinstalldir>/genomes/Hsapiens/hg19/rnaseq/ericscript/ericscript_db_homosapiens_ensembl73
    env: /<bcbioinstalldir>/anaconda/envs/samtools0

The following is the test output. The code is from ericscript branch. In case you don't know already :): You need to clone, build and install this branch to test it.

pytest ./integration/rnaseq/test_ericscript.py

========================================================================================================================================================================== test session starts ===========================================================================================================================================================================
platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/user/github/bcbio-nextgen, inifile:
plugins: mock-1.6.3, cov-2.5.1
collected 2 items

integration/rnaseq/test_ericscript.py FF

================================================================================================================================================================================ FAILURES ================================================================================================================================================================================
________________________________________________________________________________________________________________________________________________________ test_detect_fusions_with_ericscipt_without_disambiguate _________________________________________________________________________________________________________________________________________________________

install_test_files = None, data_dir = '/home/user/github/bcbio-nextgen/tests/data/automated', setup_logging = None

    @pytest.mark.install_required
    def test_detect_fusions_with_ericscipt_without_disambiguate(
            install_test_files, data_dir, setup_logging):
        """Run gene fusion analysis on trimmed pair-end reads with EricScript.
           Requires installation of EricScript and its reference data.
        """
        with make_workdir() as work_dir:
            sample_config = create_sample_config(
                data_dir, work_dir, disambiguate=False)
>           ericscript.run(sample_config)

integration/rnaseq/test_ericscript.py:37:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../bcbio/rnaseq/ericscript.py:27: in run
    run_ericscript(config, input_files)
../bcbio/rnaseq/ericscript.py:53: in run_ericscript
    build_bwa_index_if_absent(es_config, sample_config)
../bcbio/rnaseq/ericscript.py:62: in build_bwa_index_if_absent
    if not os.path.exists(es_config.reference_index):
../bcbio/rnaseq/ericscript.py:146: in reference_index
    return os.path.join(self._db_location, self._REF_INDEX)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = None, p = ('data/homo_sapiens/allseq.fa.bwt',), path = None, b = 'data/homo_sapiens/allseq.fa.bwt'

    def join(a, *p):
        """Join two or more pathname components, inserting '/' as needed.
        If any component is an absolute path, all previous path components
        will be discarded.  An empty last part will result in a path that
        ends with a separator."""
        path = a
        for b in p:
            if b.startswith('/'):
                path = b
>           elif path == '' or path.endswith('/'):
E           AttributeError: 'NoneType' object has no attribute 'endswith'

/bcbio/anaconda/lib/python2.7/posixpath.py:70: AttributeError
__________________________________________________________________________________________________________________________________________________________ test_detect_fusions_with_ericscipt_with_disambiguate __________________________________________________________________________________________________________________________________________________________

install_test_files = None, data_dir = '/home/user/github/bcbio-nextgen/tests/data/automated', setup_logging = None

    @pytest.mark.install_required
    def test_detect_fusions_with_ericscipt_with_disambiguate(
            install_test_files, data_dir, setup_logging):
        """Run gene fusion analysis on disambiguated reads with EricScript.
           Requires installation of EricScript and its reference data.
        """
        with make_workdir() as work_dir:
            sample_config = create_sample_config(
                data_dir, work_dir, disambiguate=True)
>           ericscript.run(sample_config)

integration/rnaseq/test_ericscript.py:50:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../bcbio/rnaseq/ericscript.py:27: in run
    run_ericscript(config, input_files)
../bcbio/rnaseq/ericscript.py:53: in run_ericscript
    build_bwa_index_if_absent(es_config, sample_config)
../bcbio/rnaseq/ericscript.py:62: in build_bwa_index_if_absent
    if not os.path.exists(es_config.reference_index):
../bcbio/rnaseq/ericscript.py:146: in reference_index
    return os.path.join(self._db_location, self._REF_INDEX)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = None, p = ('data/homo_sapiens/allseq.fa.bwt',), path = None, b = 'data/homo_sapiens/allseq.fa.bwt'

    def join(a, *p):
        """Join two or more pathname components, inserting '/' as needed.
        If any component is an absolute path, all previous path components
        will be discarded.  An empty last part will result in a path that
        ends with a separator."""
        path = a
        for b in p:
            if b.startswith('/'):
                path = b
>           elif path == '' or path.endswith('/'):
E           AttributeError: 'NoneType' object has no attribute 'endswith'

/bcbio/anaconda/lib/python2.7/posixpath.py:70: AttributeError
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[2017-11-02T17:00Z] INFO: Converting disambiguated reads to fastq...
[2017-11-02T17:00Z] DEBUG: Picard: SamToFastq
[2017-11-02T17:00Z] DEBUG: unset JAVA_HOME && export PATH=/bcbio/anaconda/bin:$PATH && picard -Xms750m -Xmx2g -XX:+UseSerialGC SamToFastq INPUT=/home/user/github/bcbio-nextgen/tests/data/automated/../fusion/input/Test1.nsorted.human.sorted.bam FASTQ=/home/user/github/bcbio-nextgen/tests/test_automated_output/
bcbiotx/tmpxwz8sv/Test1.nsorted.human.sorted_1.fastq TMP_DIR=/home/user/github/bcbio-nextgen/tests/test_automated_output/bcbiotx/tmpFycsXy SECOND_END_FASTQ=/home/user/github/bcbio-nextgen/tests/test_automated_output/fastq_convert/Test1.nsorted.human.sorted_2.fastq VALIDATION_STRINGENCY=SILENT
[2017-11-02T17:00Z] DEBUG: 18:00:20.482 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/bcbio/anaconda/share/picard-2.14-0/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[2017-11-02T17:00Z] DEBUG: [Thu Nov 02 18:00:20 CET 2017] SamToFastq INPUT=/home/user/github/bcbio-nextgen/tests/data/automated/../fusion/input/Test1.nsorted.human.sorted.bam FASTQ=/home/user/github/bcbio-nextgen/tests/test_automated_output/bcbiotx/tmpxwz8sv/Test1.nsorted.human.sorted_1.fastq SECOND_END_FASTQ=/home/user/github/bcbio-nextgen/tests/test_automated_output/fastq_convert/Test1.nsorted.human.sorted_2.fastq TMP_DIR=[/home/user/github/bcbio-nextgen/tests/test_automated_output/bcbiotx/tmpFycsXy] VALIDATION_STRINGENCY=SILENT    OUTPUT_PER_RG=false COMPRESS_OUTPUTS_PER_RG=false RG_TAG=PU RE_REVERSE=true INTERLEAVE=false INCLUDE_NON_PF_
READS=false CLIPPING_MIN_LENGTH=0 READ1_TRIM=0 READ2_TRIM=0 INCLUDE_NON_PRIMARY_ALIGNMENTS=false VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[2017-11-02T17:00Z] DEBUG: [Thu Nov 02 18:00:20 CET 2017] Executing as vql@cybertron on Linux 4.4.0-97-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_121-b15; Deflater: Intel; Inflater: Intel; Picard version: 2.14-SNAPSHOT
[2017-11-02T17:00Z] DEBUG: [Thu Nov 02 18:00:22 CET 2017] picard.sam.SamToFastq done. Elapsed time: 0.03 minutes.
[2017-11-02T17:00Z] DEBUG: Runtime.totalMemory()=760217600
======================================================================================================================================================================== 2 failed in 4.91 seconds ========================================================================================================================================================================