Comments (8)
maybe avoiding in the FEATURE object
self._load_qualifier_translations(Feature.DEFAULT_QUALIFIER_TRANSLATION_FILE + translation_files)
self._load_feature_translations(Feature.DEFAULT_FEATURE_TRANSLATION_FILE)
for every feature but only once at the beginning will save lot of time.
from emblmygff3.
Hello,
Thanks for the software, it is quite useful. I really appreciate it.
I'll start this by saying I am likely an edge-case user. I'm converting an 11 GB gff3 for a poison frog with a 6.8 Gbp genome. Currently it is 42 hours in and still writing to a flat file that is 4.4 GB. Is there an explicit flag for multithreading? I'm sure I'll have to do this again, so I'm wondering if there is a way to speed up the process.
Thanks!
from emblmygff3.
Are you using version2?
You can try version3 which is in the branch feature/speedup but not all features are implemented. This version is multithreaded.
Otherwise you can split you files into chunks and run them in parallel using --locus_numbering_start to avoid having similar loci name.
from emblmygff3.
Hi @AdamStuckert, I have made an update of the version 2 to make it faster (it is on branch "speed"). It is not multi threaded but might be 10X faster. Could you give a try on your huge file?
from emblmygff3.
Thanks very much @Juke34! I'll install it and test it out today. I'll let you know how it does in the near future.
from emblmygff3.
Hi @AdamStuckert, did you succeed to run the tool from the speed branch? Did you see any improvement in term of time?
from emblmygff3.
Hi,
Sorry for the delay. I meant to test this out and then gestures vaguely at the USA. Anyway, it ran in ~24 hours this time with the same genome assembly and annotation file, which is a really dramatic improvement in speed. Thanks!
from emblmygff3.
Thank you for your feedback. I have seen better performance on small files but it is still ~2 times faster on your huge file.
I will merge the branch and release a version 2.1 soon.
To resume:
- Performance has been improved in version 1.2.5 with a better way to deal with the sequence
- Performance has been improved in version 2.1 with a better way to deal with translation_files (loaded only once at the beginning then using a dictionary)
- Performance are even better in version 3 where everything has been redesigned. It is faster and parallelized. But not all options from previous versions are implemented yet, only core set of parameters. It is not yet released and can be found in branch
feature/speedup
from emblmygff3.
Related Issues (20)
- How to add in comment or CC line HOT 1
- Webin-CLI validation failing due to duplicated feature locations in EMBLmyGFF3 flat file HOT 2
- thank you HOT 1
- ImportError Bio.Alphabet error HOT 1
- What the option "-a accesion" parameter should to be set(means:What type?) HOT 3
- TypeError: read() takes 1 positional argument but 2 were given? HOT 3
- attribute formats broken across several lines HOT 2
- Use of example data HOT 2
- Contigs are disorganised in the EMBL file HOT 10
- Warning qualifier unknown db_xref HOT 1
- Translation problem HOT 2
- Reporting the line number of the problematic input GFF3 files when parsing error is triggered HOT 2
- BioPython 1.81 installs via conda, but needs an older version HOT 2
- if I can specify certain python source during python setup.py install HOT 2
- Not for ENA submission: Sequence too short
- Installation issue EMBLmyGFF3 & python version requirements
- Translation when circular genome and ORF in the cut of the assembly HOT 1
- unexpected keyword argument 'strand' HOT 3
- Gene sorting compared to fasta HOT 4
- Bug when specifying -g mitochrondrion (or plastid)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emblmygff3.