estebanpw / chromeister Goto Github PK
View Code? Open in Web Editor NEWA dotplot generator for large chromosomes
License: GNU General Public License v3.0
A dotplot generator for large chromosomes
License: GNU General Public License v3.0
Hi There,
I followed the install instructions and I encountered the following error on Ubuntu 22.04.1 LTS:
After this:
cd chromeister && make all -C src/ && python3 -m venv chromeisterenv && source chromeisterenv/bin/activate && pip install -r src/requirements.txt
I have the following error:
/tmp/pip-build-env-uj14l2ll/overlay/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:242:9: error: 'PyTuple_GET_SIZE' was not declared in this scope
if (PyTuple_GET_SIZE(value) != 3) {
^~~~~~~~~~~~~~~~
/tmp/pip-build-env-uj14l2ll/overlay/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:242:9: note: suggested alternative: 'PyTuple_GetSlice'
if (PyTuple_GET_SIZE(value) != 3) {
^~~~~~~~~~~~~~~~
PyTuple_GetSlice
/tmp/pip-build-env-uj14l2ll/overlay/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:245:13: error: 'PyTuple_GET_ITEM' was not declared in this scope
title = PyTuple_GET_ITEM(value, 2);
^~~~~~~~~~~~~~~~
/tmp/pip-build-env-uj14l2ll/overlay/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:245:13: note: suggested alternative: 'PyArray_GETITEM'
title = PyTuple_GET_ITEM(value, 2);
^~~~~~~~~~~~~~~~
PyArray_GETITEM
gmake[2]: *** [modules/python3/CMakeFiles/opencv_python3.dir/build.make:76: modules/python3/CMakeFiles/opencv_python3.dir/__/src2/cv2.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:2077: modules/python3/CMakeFiles/opencv_python3.dir/all] Error 2
gmake: *** [Makefile:166: all] Error 2
Traceback (most recent call last):
File "/tmp/pip-build-env-uj14l2ll/overlay/lib/python3.7/site-packages/skbuild/setuptools_wrap.py", line 640, in setup
cmkr.make(make_args, install_target=cmake_install_target, env=env)
File "/tmp/pip-build-env-uj14l2ll/overlay/lib/python3.7/site-packages/skbuild/cmaker.py", line 672, in make
self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
File "/tmp/pip-build-env-uj14l2ll/overlay/lib/python3.7/site-packages/skbuild/cmaker.py", line 704, in make_impl
"An error occurred while building with CMake.\n"
An error occurred while building with CMake.
Command:
cmake --build . --target install --config Release --
Install target:
install
Source directory:
/tmp/pip-install-pjy3pvde/opencv-python
Working directory:
/tmp/pip-install-pjy3pvde/opencv-python/_skbuild/linux-x86_64-3.7/cmake-build
Please check the install target is valid and see CMake's output for more information.
----------------------------------------
ERROR: Failed building wheel for opencv-python
Running setup.py clean for opencv-python
Failed to build opencv-python
ERROR: Could not build wheels for opencv-python which use PEP 517 and cannot be installed directly
I checked the requirements for the program and I workaround the following conda recipe for Ubuntu 22.04:
cd --
git clone https://github.com/estebanpw/chromeister.git
cd chromeister
make all -C src/
conda create --name chromeister python=3.6.13 # create environment
conda activate chromeister # load environment
# Install packages
conda install -c bioconda -y scikit-build
conda install -c anaconda -y cycler
conda install -c anaconda -y kiwisolver
conda install -c bioconda -y numpy
pip install opencv-python
conda install -c bioconda -y Pillow
conda install -c bioconda -y pyparsing
conda install -c bioconda -y python-dateutil
conda install -c bioconda -y six
In this manner, as an example, I can call chromeister as follows:
conda activate chromeister
/home/carlos/chromeister/bin/CHROMEISTER -query GRCm39.genome.fa -db Mus_musculus.GRCm39.dna.toplevel.fa -out dotplot.mat && Rscript /home/carlos/chromeister/bin/compute_score.R dotplot.mat 1000
and the program is working as follows:
[INFO] Generating a 1000x1000 matrix
[INFO] Loading database
99%...[INFO] Database loaded and of length 2728222451.
[INFO] Ratios: Q [2.728222e+06] D [2.728222e+06]. Lenghts: Q [2728222451] D [2728222451]
[INFO] Pixel size: Q [3.665390e-07] D [3.665390e-07].
[INFO] Computing absolute hit numbers.
19%...
79%...
99%...Scanning hits table.
99%...
[INFO] Query length 2728222451.
[INFO] Writing matrix.
[INFO] Found 2234272 unique hits for z = 4.
0
My question is, can I just use the program in this way, or I do need to work on the python-opencv error?
Cheers,
Carlos
Hi,
I am trying to use chrmister for comapring genomes. All these genomes have ~1900 scaffolds. At first for the first pair the run_and_plot_chromeister.sh script worked perfectly well and I got really nice plots at both 1000 and 2000 dims.
Then for the next pair it is giving me the errors. Here is the log:
/apps/chromeister/1.5a/chromeister/bin/run_and_plot_chromeister.sh: line 31: /apps/chromeister/1.5a/chromeister/bin/../chromeisterenv/bin/activate: No such $
Traceback (most recent call last):
File "/apps/chromeister/1.5a/chromeister/bin/detect_events.py", line 15, in
readheader = open(sys.argv[1], "r")
FileNotFoundError: [Errno 2] No such file or directory: 'EW18.fasta-GM1.fasta.mat.raw.txt'
/apps/chromeister/1.5a/chromeister/bin/run_and_plot_chromeister.sh: line 33: deactivate: command not found
Mon Jun 27 03:38:52 EDT 2022
Command in the slurm script is:
module load chromeister/1.5a
module load R/4.2
run_and_plot_chromeister.sh EW18.fasta GM1.fasta 32 1000 4 grid
date
Hello Esteban,
I'm experiencing hard times trying to modify the R Script to get vectorized plots to post-edit them. I'm getting over-dimensioned plots and I have no idea about what is going on despite I think is the same code after change the format. Could you help me?
Thank you in advance!
Hello Esteban,
Many thanks for developing such useful tools.
1). I would like to use chromeister to check inversion and their breakpoint (first figure). Does chromeister can highlight or report the breakpoint of inversion in the png?
2). When I use chromeister to check the small inversion event, I find the figure not like dotpoint(second figure), but dotline. it's because of k-mer size?
Thanks.
best,
Lin
Hi there,
Thank you for developing chromeister. I have used the software for a series of genome comparisons. Firstly, I compared two versions of genome assemblies from the same species. One of them is too fragmentated (~7000 scaffolds) and the other one is chromosome-level assembly with 16 large scaffolds (chr). I expected to see a small score for this comparison but the score value was 0.73. As you mentioned in your manuscript, the scores close to 0 indicates the exact same sequences and 1 indicates absolutely no similarity (if I am right). I wonder if this could be the results of fragmentation in the first genome and if you have any suggestion to improve that. I have also compared the genome assembly of this species (second one) with two other related species and the scores were 0.1 and 0.26, which were reasonable. Both of these species have a chromosome-level assemblies. I would appreciate it if you could guide me in this regard.
Best,
Karim
Hello,
I ran Chromeister and I get the following error. Could you please help me? Thanks!
./CHROMEISTER -query query.fa -db ref.fa -out dotplot.mat && Rscript compute_score.R dotplot.mat 1000
[INFO] Generating a 1000x1000 matrix
[INFO] Loading database
100%...[INFO] Database loaded and of length 557480303.
[INFO] Ratios: Q [6.709421e+05] D [5.574803e+05]. Lenghts: Q [670942087] D [557480303]
[INFO] Pixel size: Q [1.490442e-06] D [1.793785e-06].
[INFO] Computing absolute hit numbers.
100%...Scanning hits table.
100%...
[INFO] Query length 670942087.
[INFO] Writing matrix.
[INFO] Found 299180 unique hits for z = 4.
[1] 0.996
Error in axis(1, tick = FALSE, labels = seq_x_labels, at = seq_x_ticks) :
no locations are finite
Execution halted
Hi Esteban,
I'm interested in pulling the specific chromosomal coordinates identified in the events file after running run_and_plot_chromeister.sh. I believe they are reported globally across the genome (the genome has 21 chromosomes in the fasta) in this file, nor does the file specify the chromosome from which they are derived. What is the best way to covert global coordinate to local chromosomal coordinates for the events identified?
Thanks in advance,
Jeff
Hi,
I was running CHROMEISTER on a plant genome against itself (~1 Gbp) where I expected to see some synteny between chromosomes, but did not see any. I then tested CHROMEISTER on a chloroplast genome (which has two inverted repeat regions) and again saw nothing. I then decided to copy parts of the cp chromosome into test contigs - both forward and reverse complement of ~5kb. This too showed no synteny, just a dot plot with three gaps (fig1 below). What is the minimum size of a block where you would expect to see synteny (I would have thought 5kb would be large enough?).
I then repeated the test with the whole test sequence reverse complemented and as added as another chromosome:
CHROMEISTER -dimension 2000 -kmer 16 -query cptest2.fasta -db cptest2.fasta -out cpdotplot.mat && Rscript /g/data/nm31/bin/chromeister/bin/compute_score.R cpdotplot.mat 2000
[INFO] Generating a 1000x1000 matrix
[INFO] Loading database
100%...[INFO] Database loaded and of length 374340.
[INFO] Ratios: Q [3.743400e+02] D [3.743400e+02]. Lenghts: Q [374340] D [374340]
[INFO] Pixel size: Q [2.671368e-03] D [2.671368e-03].
[INFO] Computing absolute hit numbers.
100%...Scanning hits table.
99%...
[INFO] Query length 374340.
[INFO] Writing matrix.
[INFO] Found 0 unique hits for z = 4.
0.996
..and this produced a 'grey' dotplot, fig2.
test data:
So something is not working but I am not getting any errors.
Thanks,
Theo
Hi,
I'm working on the comparison of whole genome sequence for different maize lines.
The genome is large, ~ 2GB per line, and Mummer took me a long tome for each pairwise comparison. I found that chromeister is very fast and I want to apply it to all my comparison.
I need an output format similar as the output by show-coords from Mummer. The output should includes start and end position on the query and target. However, I only got a coordinate as the plot X-Y axis using gecko from chromeister output.
Could you help me to figure out how to get the coordinates based on each chromosome?
Best,
Jing
Hi.
Can chromeister be used for comparing whole genome of two mammals such as human and chimp.
if we need to specify/modify certain parameters please mention those as well..
thank you.
mohit
In addition, there is another small issue where the removal of "index-refseq-qryseq.csv" is not successful, which cause an error.
It seems that that is caused by the directory /path issue.
error information:
Launching... /home/xuewen/Downloads/chromeister-master/bin/index_chromeister_solo.sh . index-refseq-qryseq.csv /home/xuewen/Downloads/chromeister-master/refseq /home/xuewen/Downloads/chromeister-master/qryseq
rm: cannot remove 'index-refseq-qryseq.csv': No such file or directory
$scriptdir/CHROMEISTER -query aradu.K30065.gnm1.chr01.fa -db aradu.K30060.gnm1.chr01.fa -out dotplot.mat -dimension 2000
output: 7.7M
dotplot.mat
error:
$ Rscript $scriptdir/compute_score.R dotplot.mat 2000
Error in file(con, "r") : cannot open the connection
Calls: unlist -> readLines -> file
In addition: Warning message:
In file(con, "r") :
cannot open file 'dotplot.mat.csv': No such file or directory
Execution halted
##it seems that the output "dotplot.mat.csv " is missing
Hi
Is it possible to sort the chromeister plot according to the reference coordinates? I found a thread discussing the same using Mauve but I am dealing with a huge genome and It doesn't work.
Thanks
Dear Esteban,
I used chromeister tool to compare two varieties of olive plants (Each one was over 3 Gbp ) using the command
"./CHROMEISTER -query Genome1.fa -db Genome2.fa -out dotplot.mat -diffuse 1 && Rscript compute_score-nogrid.R dotplot.mat 1000"
Although I got the percentage of similarity the plot was messy enough
Then I run
/CHROMEISTER -query Genome1.fa -db Genome2.fa -out dotplot.mat -diffuse 1 -dimension 2000 && Rscript compute_score-nogrid.R dotplot.mat 2000 But the plot was messy again...
The genomes that I compare have hundreds of contigs, and thus the figure looks messy. Could you provide some instructions on tuning the parameters to generate a better plot ?
Thanks a lot,
Mary
Just found the chromeister galaxy tool https://toolshed.g2.bx.psu.edu/repository?repository_id=908b5504c1930f12 and noticed a few possibilities to improve it. I would suggest to
If you are interested I'm happy to help.
I installed the chromeister using conda.
I run the command
CHROMEISTER -query GENOME1.fasta -db GENOME2.fasta -out dotplot.mat -dimension 2000 && Rscript compute_score.R dotplot.mat 2000
but at last I faced issue of
Fatal error: cannot open file 'compute_score.R': No such file or directory
I will be thankful for help.
Hi There,
I am going to compare two large genome assemblies (each ~ 2.6 Gb). I am interested to use allVsAll.sh for this purpose but I have two concerns: 1) I did not separate chromosomes and there are two whole genomes in my genome directory. I wonder if this is a correct way to do such analysis. 2) I would like to use parallel analysis and you mentioned that it can be possible by re-issuing the command but I want to submit the job to server (slurm job), so I wonder how I can handle that.
Here is the my command:
allVsAll.sh /chromeister/genomes/ fasta dim 2000 kmer 32
Thank you for your support.
Karim
If there are too many sequences in a multifasta (e.g. contigs or scaffolds) the grid will completely eclipse the dotplot.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.