chileanvirtualobservatory / acalib Goto Github PK
View Code? Open in Web Editor NEWAdvanced Computing for Astronomy Library
Home Page: http://acalib.readthedocs.io/en/latest/
License: GNU General Public License v3.0
Advanced Computing for Astronomy Library
Home Page: http://acalib.readthedocs.io/en/latest/
License: GNU General Public License v3.0
The WCS pix2world conversion on astropy CORRECTLY turn around the clock when you arrive to a zero limit. For example, if the object is in alpha=0.0 center, then all the pixels to the right will be like delta_something, while pixels to the left will be 365 - delta_something, rather than simply the negative number. This is a complex issue that we must address here!
The name of the class "Indexing" is too general. We should change the name to something more clear, and refer in the documentation to the paper and to the original code. Probably we should discuss the name with @marcelomendoza
Getting errors when running Clumping Algorithms acalib.Fellwalker
, acalib.ClumpFind
in python2, the error comes from pycupid (ChileanVirtualObservatory/pycupid#4)
It returns a numpy array, while having all the information to return a NNData with the proper WCS that will display correctly in the visualize() function.
Using http://astrojs.github.io/sampjs/ we should be able to send NDData and Astropy tables from local apps or from a server (e.g. JupyterHub) to the SAMP Hub.
Need the commands and procedures to install acalib devel in a conda enviroment. The idea is that a script can do everything so it must be clear and automatic.
Someone should pick the work of @teohoch and integrate it to the devel branch. The main idea is to remove the sqlite database and use the elastic slap service with cache. The code is there
Prepare system for installing ACALIB:
Currently we are using wcsaxes package for 2D visualization. We can do better by using APLpy. An example can be found here https://github.com/ChileanVirtualObservatory/notebooks/blob/master/course/II%20-%20Accesing%20Data.ipynb
In my latest refactoring (yep... I do one each month it seems), I've removed the nddata and table support from the core. This was done because the core should be used by the UPI and Algorithms, so we can put all the Astropy data structures (Table and NDData) in these other packages rather than in the core.
I have not done this for all the functions. Therefore,
There is no algorithms/stacking.py
file, and no inherited class from Algorithm that implement what is already supposed to be working in the core. Remember to document this class
I think we need a more mature library for a 1.0 release.
Before merging the pull request #54 I tested the algorithms, but not all the UPI functions.
Now im trying to use acalib.moment0
and I get AttributeError: 'numpy.float64' object has no attribute 'wcs'
Maybe is the @support_nddata
decorator or something else.
The code done by @vansimonsen needs to be reviewed and corrected if needed. Also a notebook with a concrete example should be provided (in the notebooks repository). This is classified as a bug because we already found some problems with the alignments of the images.
We need to remove this branch.
The documentation of Algorithm is incomplete. This should include a sphinx entry with the general usage of algorithms in ACALib
Provide a Jupyter-friendy visualization of a NDData in 3D (interactive)
When using loadFITS_PrimmaryOnly()
with big fits files (e.g. ALMA01002186_00_00_00.fits 1.8G) the function crashes with a MemoryError exception in the line:
hduobject.data = (hduobject.data*bscale) + bzero
It create the fits with a duplicated image (container.primary and first image of container.image array)
We need to protect our code (requirement from our funding agency), so every file need to have the GPL license in the header, plus the GPL files required in the main directory.
An example of such header can be found here
The container is meant to read a fits in a generic way, yet I guess this is impossible, because there are too many variations of them. However, what we can do is to provide only the HDU_to_fits() function. This way, the fits reading is responsibility of the user (there are a lot of tutorials!), and we only do the transformation to NDData. What do you think?
The script is just for building the wheel for now, we need to install the wheel and run the tests.
performing a python mathematical operation over a numpy array is terribly SLOW:
In [1]: import timeit
In [2]: import numpy as np
In [3]: a=np.random.random((10,100))
In [4]: f1=lambda :a.sum()
In [5]: f2=lambda : sum(sum(a))
In [6]: %timeit f1()
100000 loops, best of 3: 2.5 µs per loop
In [7]: %timeit f2()
10000 loops, best of 3: 33.4 µs per loop
So... please use numpy!
The core subpackage does not need to be documented in sphinx, but all functions must include a brief description. Currently half of them have it.
What do you think?
The https://github.com/ChileanVirtualObservatory/SpeLAR repository includes an algorithm for labeling spectral lines using association rules. We need to integrate this code into ACALib.
I had a chat with Tim Jenness (maintainer of AST and pyAst,, currently working for the LSST), and we have to decide what we will use: the astropy.wcs and the contributed package gwcs, or AST/pyAst. AST supports a lot of fits file formats, and have a robust wcs management (AST is basically for that), while the astropy alternative is in open development strategy and more maintained. I've seen @msanchezc putting the AST library in the devel branch for supporting GaussClump. Why this is needed?. I really dislike AST to support things like AstKeyMaps, but if GaussClump is already using stuff from FITS/WCS, we should consider AST seriously. Comments?
The clumFind algorithm continue to crash in some (apparently random) cases. We need to debug this issue.
There is no documentation of ClumpFind and FellWalker algorithm classes (Sphinx). It will be easier to refer the user to the cupid official documentation rather than explaining the parameters ourselves. The documentation of GaussClump is deferred to the devel branch.
CASA-generated files include stokes and we drop it simply by position. We must free ACALib from hardcoded dimensions (i.e., arr[0] is freq), because that depends on the FITS metada. We need to clarify this, and make sure that we can use ACALib on "any" fits, not only CASA-generated ones.
After compiling pycupid and _morph got an error importing the libraries.
ImportError: /home/tronco/Desktop/acalib/acalib/core/_morph/morph.so: undefined symbol: _Py_ZeroStruct
This happens only in Python 3, with Python 2 i got no problem importing the libraries.
We should use a pixelset structure. The functions like spectra works for a bounding box, which is correct, but we need to support a "pixel set" also.
The https://github.com/ChileanVirtualObservatory/display repository include code for automatic detection of spectral lines using training data (ML). We need to integrate this to ACALib
The UPI includes a noise_level() function that is a NDData wrapper of core's rms() function. Here we should check for background estimation rather that simply computing the rms. We might offer the rms as one method of the background estimation (maybe the default).
We have defined a new repo:
https://github.com/ChileanVirtualObservatory/pycupid
where pyCupid is now.
We need to:
The pull request #30 is accepted, but it lacks from the .config interface as the other algorithms.
I installed acalib with the following commands:
su - $USER -c "git clone https://github.com/ChileanVirtualObservatory/acalib.git"
su - $USER -c "cd /home/$USER/acalib && git checkout devel && /home/$USER/conda/bin/pip install -e ."
and I get this error in a jupyterhub-notebook when import:
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-72aa9cf42e28> in <module>()
----> 1 import acalib
~/acalib/acalib/__init__.py in <module>()
1 from __future__ import absolute_import
2
----> 3 from .algorithms import *
4 from .io import *
5 from .upi import *
~/acalib/acalib/algorithms/__init__.py in <module>()
2 from .clumpfind import ClumpFind
3 from .gms import GMS
----> 4 from .roise import RoiSE
5 from .stacking import Stacking
6 from .represent import HRep
ImportError: cannot import name 'RoiSE'
CUPID does not only return Clump Assignment Arrays, but there are other several data structures that would be very useful to return to the user. This might need some cython support in the pyCupid project.
All the Mayavi stuff is heavily hardcoded: graph.py include de axes and only works for objects with velocities.
The UPI should have its own and very detailed sphinx documentation, because is the entrypoint for the astronomer.
run_test.py
throws the following error sometimes:
======================================================================
FAIL: test_morph (core.test_analysis.TestAnalysis)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/apawn/Documents/repos/acalib/testing/core/test_analysis.py", line 72, in test_morph
np.testing.assert_almost_equal(boxing,result_boxing)
File "/home/apawn/Music/miniconda3/envs/acalibenv/lib/python3.6/site-packages/numpy/testing/utils.py", line 563, in assert_almost_equal
return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/home/apawn/Music/miniconda3/envs/acalibenv/lib/python3.6/site-packages/numpy/testing/utils.py", line 962, in assert_array_almost_equal
precision=decimal)
File "/home/apawn/Music/miniconda3/envs/acalibenv/lib/python3.6/site-packages/numpy/testing/utils.py", line 778, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 7 decimals
(mismatch 10.0%)
x: array([ 6.9413918e-310, 1.0000000e+000, 1.0000000e+000,
1.0000000e+000, 1.0000000e+000, 1.0000000e+000,
1.0000000e+000, 1.0000000e+000, 1.0000000e+000,
8.4286332e+174])
y: array([ 6.9247206e-310, 1.0000000e+000, 1.0000000e+000,
1.0000000e+000, 1.0000000e+000, 1.0000000e+000,
1.0000000e+000, 1.0000000e+000, 1.0000000e+000,
6.9246513e-310])
Python 2 and Python 3 have different import support.
from . import x
does not work well in Python 2.7.
We should reorganise imports for hiding .core
and only showing the UPI, Algorithms and I/O
The 3D visualization is out of the question for ACALib 1.0. But in the devel branch is commented a code for performing an animation of cubes by frequency. We must have a slider version (similar to CASA or dendroviewer) to visualize a cube.
"""
Extracts properties information of the astronomical data cube.
Parameters
----------
img : numpy.ndarray
Astronomical data cube.
Returns
-------
result : dict
Dictionary with properties of the image: *centroid*, *major*, *minor*, *ratio*, *angle*, *area*, *img*, *clr*, *label*, *orig*.
"""
Doens't seem to be enough for this function that does a lot of things. Maybe describe each value of the output dictionary?
Since astropy 1.2.1, the NDData package brings a lot of new features, like mixings, slicing and it says I/O operations (but I do not see them in the code). The master plan of this NDData is
https://github.com/astropy/astropy-APEs/blob/master/APE7.rst
We need to check well this and use it correctly (inherit) in order to provide nice compatibility.
I've developed a few jovial stylers for arrays and tuples with quantities. These are at io/__init__.py
right now. We should find a place for this, and decide if goes into the master or the devel branch
FITS archives with some NaN pixels in a working image can't be read.
ex: ALMA01000287_00_00_00.fits failed because of NaN
This algorithm works, but we believe it can be largely improved in terms of computational efficiency.
We are currently using the NDData interface for creating and loading data. This should be ported to the new NDDataRef interface, including algorithms, loading and saving using the standard I/O astropy interface and implementing the slicing methods.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.