Comments (12)
Could you try running the unit test (cd MIDAS/test; test_midas.py)? Could you also provide me the list of commands you ran?
Thanks,
Stephen
from midas.
Hi @snayfach,
Here is the output from the unit test:
apps/MIDAS/test$ python test_midas.py
..F..F..
======================================================================
FAIL: test_help_text (__main__.MergeGenes)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_midas.py", line 92, in test_help_text
self.assertTrue(sum(self.retcodes)==0, msg=error)
AssertionError:
Failed to execute the command: merge_midas.py genes
======================================================================
FAIL: test_help_text (__main__.RunGenes)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_midas.py", line 64, in test_help_text
self.assertTrue(run(self.command)==0, msg=error)
AssertionError:
Failed to execute the command: run_midas.py genes ./sample -1 ./test.fq.gz -n 100 --species_id 57955
----------------------------------------------------------------------
Ran 8 tests in 455.129s
FAILED (failures=2)
These are the commands I ran:
run_midas.py species ERR866577_species -1 ERR866577_1.fastq.gz -2 ERR866577_2.fastq.gz -t 20
run_midas.py snps ERR866577_species --species_id 57327 -1 ERR866577_1.fastq.gz -2 ERR866577_2.fastq.gz -t 20
run_midas.py snps ERR866577_species -1 ERR866577_1.fastq.gz -2 ERR866577_2.fastq.gz -t 20
Both snps
commands failed with the above bowtie error.
Thanks for the help,
Daryl
from midas.
Hi Daryl,
It looks like the unit test is failing. Could you send me the output from running:
cd MIDAS/test
run_midas.py genes ./sample -1 ./test.fq.gz -n 100 --species_id 57955
from midas.
Hi Stephen,
Sorry for a mix up, I realized I didn't have my python ENV activated and our system pysam version is old, and that is why it was failing.
After I realized this, and activated the ENV, the test_midas.py
runs fine.
apps/MIDAS/test$ python test_midas.py
........
----------------------------------------------------------------------
Ran 8 tests in 506.385s
OK
Despite this, I'm still getting the original error bowtie error for the snps
command (after species
has worked fine):
Error encountered executing:
apps/MIDAS/bin/Linux/bowtie2-build ERR866577_test/snps/temp/genomes.fa ERR866577_test/snps/temp/genomes
Error message:
Warning: Empty fasta file: 'ERR866577_test/snps/temp/genomes.fa'
Warning: All fasta inputs were empty
Error: Encountered internal Bowtie 2 exception (#1)
Command: bowtie2-build --wrapper basic-0 ERR866577_test/snps/temp/genomes.fa ERR866577_test/snps/temp/genomes
The odd thing is is that ERR866577_test/snps/temp/genomes.fa
is actually not empty. It appears to be the correct fasta files.
Update:
It appears that directly entering the command works.
apps/MIDAS/bin/Linux/bowtie2-build --wrapper basic-0 ERR866577_snps/snps/temp/genomes.fa ERR866577_snps/snps/temp/genomes
Produces relevant genomes.*.bt2
files
from midas.
It's strange that the unit test works but you're getting an error on your own data. Could you double check that this works:
cd MIDAS/test
run_midas.py snps ./sample -1 ./test.fq.gz -n 100 --species_id 57955
Are you trying to run run_midas.py snps
twice at the same time with the same output directory (unclear from reading your earlier post)? If so, then this is probably the issue.
from midas.
Test data works fine:
apps/MIDAS/test$ run_midas.py snps ./sample -1 ./test.fq.gz -n 100 --species_id 57955
MIDAS: Metagenomic Intra-species Diversity Analysis System
version 1.0.0; github.com/snayfach/MIDAS
Copyright (C) 2015-2016 Stephen Nayfach
Freely distributed under the GNU General Public License (GPLv3)
===========Parameters===========
Script: run_midas.py snps
Output directory: ./sample
Remove temporary files: False
Pipeline options:
-build bowtie2 database of genomes
-align reads to bowtie2 genome database
-use samtools to generate pileups and call SNPs
Database options:
-include specified species id(s): ['57955']
Read alignment options:
-input reads (1st mate): ./test.fq.gz
-input reads (2nd mate): None
-alignment speed/sensitivity: very-sensitive
-number of reads to use from input: 100
-number of threads for database search: 1
SNP calling options:
-minimum alignment percent identity: 94.0
-minimum mapping quality score: 20
-minimum base quality score: 30
-minimum read quality score: 20
-trim 0 base-pairs from read-tails
Building database of representative genomes
total genomes: 1
total contigs: 1
total base-pairs: 5163301
0.16 minutes
0.12 Gb maximum memory
Mapping reads to representative genomes
0.01 minutes
0.12 Gb maximum memory
Running mpileup
0.07 minutes
0.12 Gb maximum memory
Formatting output
3.14 minutes
1.09 Gb maximum memory
My data using exact same command structure:
run_midas.py snps ./test -1 ./ERR866577_1.fastq.gz -2 ./ERR866577_2.fastq.gz -n 100 --species_id 57327
MIDAS: Metagenomic Intra-species Diversity Analysis System
version 1.0.0; github.com/snayfach/MIDAS
Copyright (C) 2015-2016 Stephen Nayfach
Freely distributed under the GNU General Public License (GPLv3)
===========Parameters===========
Script: run_midas.py snps
Output directory: ./test
Remove temporary files: False
Pipeline options:
-build bowtie2 database of genomes
-align reads to bowtie2 genome database
-use samtools to generate pileups and call SNPs
Database options:
-include specified species id(s): ['57327']
Read alignment options:
-input reads (1st mate): ./ERR866577_1.fastq.gz
-input reads (2nd mate): ./ERR866577_2.fastq.gz
-alignment speed/sensitivity: very-sensitive
-number of reads to use from input: 100
-number of threads for database search: 1
SNP calling options:
-minimum alignment percent identity: 94.0
-minimum mapping quality score: 20
-minimum base quality score: 30
-minimum read quality score: 20
-trim 0 base-pairs from read-tails
Building database of representative genomes
total genomes: 1
total contigs: 107
total base-pairs: 4029290
Error encountered executing:
apps/MIDAS/bin/Linux/bowtie2-build ./test/snps/temp/genomes.fa ./test/snps/temp/genomes
Error message:
Warning: Empty fasta file: './test/snps/temp/genomes.fa'
Warning: All fasta inputs were empty
Error: Encountered internal Bowtie 2 exception (#1)
Command: bowtie2-build --wrapper basic-0 ./test/snps/temp/genomes.fa ./test/snps/temp/genomes
However at this moment running apps/MIDAS/bin/Linux/bowtie2-build ./test/snps/temp/genomes.fa ./test/snps/temp/genomes
will execute the command no problem.
Re the running two commands at the same time, I wasn't running those at the same time. Simply showing you that both of those commands failed. Didn't matter if it was the automated pipeline vs specifying a species id.
from midas.
I wonder if this is related to the species_id. Could you try running the test data on your specified species:
cd MIDAS/test
run_midas.py snps ./sample -1 ./test.fq.gz -n 100 --species_id 57327
This might also be related to file permissions. Could you try running your data, but use this specified directory for your output (replace '/path/to' with the appropriate path):
cd MIDAS/test
run_midas.py snps ./output -1 /path/to/ERR866577_1.fastq.gz -2 /path/to/ERR866577_2.fastq.gz -n 100 --species_id 57327
from midas.
I just tried running midas with --species_id 57327
and the test data and it worked fine. So the problem should be either your specified output directory or your specified input files (-1 and -2).
from midas.
It seems like it is a permissions issue. When I run the analysis from the test folder everything works. Switching to the analysis folder, it hits that error. I tried changing the permissions for the folder and still didn't work. I wonder why the python call for the script has the error whereas the call from the command line doesn't.
from midas.
Very strange. Could you try running it from your home directory where there shouldn't be any permissions issues? Does it work there?
from midas.
Sorry, was away for a few days. The pipeline works from my home directory. I'm on a large server, with multiple drives if that helps, but I've never had any problems before with installing software on one and running the analysis on another drive.
from midas.
Thanks for checking this. If you end up tracking down what's happening, I'd be happy to build in a check into the software (or a fix).
Stephen
from midas.
Related Issues (20)
- setup.py: add scripts
- test not completing sucessfully HOT 1
- Strain tracking - getting identified strain names
- test_midas.py fail for test 7 HOT 1
- Can't download database
- Building database error HOT 3
- Test 11 and 14 fails (and fixes)
- Database marker gene file "phyeco.map" has wrong genome_id label
- samtools invalid option 'f'
- build custom database error
- samtools errors HOT 1
- Samtools missing binaries HOT 3
- How can I use MIDAS to get pN/pS on a per-gene basis?
- AttributeError: Can't pickle local object 'parallel.<locals>.init_worker' HOT 1
- Question regarding speed of execution HOT 1
- test_class (__main__._07_RunSNPs) ... FAIL
- Request for information on how are marker genes chosen in custom database ? HOT 4
- Population diversity with intermediate frequency HOT 1
- cannot download the database midas_db_v1.2.tar.gz HOT 2
- no outputs for speciation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from midas.