chileanvirtualobservatory/acalib

Advanced Computing for Astronomy Library

Advanced Computing for Astronomy Library (ACALib)

Advanced Computing for Astronomy Library (ACALib) is a package with state of the art Algorithms for Astronomers.

It is intended to have a simple interface and compatibility with Jupyter Notebooks.

Documentation and Examples

The lastest documentation can be found in here. And we have some Jupyter Notebooks examples in here.

Installation

To install ACALIB just clone this repository and run:

python setup.py build install

or install using pip

pip install acalib

acalib's Issues

Wrong usage of NDData

Since astropy 1.2.1, the NDData package brings a lot of new features, like mixings, slicing and it says I/O operations (but I do not see them in the code). The master plan of this NDData is

https://github.com/astropy/astropy-APEs/blob/master/APE7.rst

We need to check well this and use it correctly (inherit) in order to provide nice compatibility.

Astrodendro Algorithm does not follow .config schema

The pull request #30 is accepted, but it lacks from the .config interface as the other algorithms.

"failed because of NaN" error.

FITS archives with some NaN pixels in a working image can't be read.
ex: ALMA01000287_00_00_00.fits failed because of NaN

UPI NDDataRef Errors

Before merging the pull request #54 I tested the algorithms, but not all the UPI functions.

Now im trying to use acalib.moment0 and I get AttributeError: 'numpy.float64' object has no attribute 'wcs'

Maybe is the @support_nddata decorator or something else.

Reorganize Imports

Python 2 and Python 3 have different import support.

from . import x does not work well in Python 2.7.

We should reorganise imports for hiding .core and only showing the UPI, Algorithms and I/O

Cython's library import error

After compiling pycupid and _morph got an error importing the libraries.

ImportError: /home/tronco/Desktop/acalib/acalib/core/_morph/morph.so: undefined symbol: _Py_ZeroStruct

This happens only in Python 3, with Python 2 i got no problem importing the libraries.

Spectra function is not UPI friendly

It returns a numpy array, while having all the information to return a NNData with the proper WCS that will display correctly in the visualize() function.

ASYDO update to elastic slap

Someone should pick the work of @teohoch and integrate it to the devel branch. The main idea is to remove the sqlite database and use the elastic slap service with cache. The code is there

3D visualization is heavily hardcoded

All the Mayavi stuff is heavily hardcoded: graph.py include de axes and only works for objects with velocities.

Algorithm documentation

The documentation of Algorithm is incomplete. This should include a sphinx entry with the general usage of algorithms in ACALib

MemoryError in loadFITS_PrimmaryOnly

When using loadFITS_PrimmaryOnly() with big fits files (e.g. ALMA01002186_00_00_00.fits 1.8G) the function crashes with a MemoryError exception in the line:
hduobject.data = (hduobject.data*bscale) + bzero

AST Library and Cupid

I had a chat with Tim Jenness (maintainer of AST and pyAst,, currently working for the LSST), and we have to decide what we will use: the astropy.wcs and the contributed package gwcs, or AST/pyAst. AST supports a lot of fits file formats, and have a robust wcs management (AST is basically for that), while the astropy alternative is in open development strategy and more maintained. I've seen @msanchezc putting the AST library in the devel branch for supporting GaussClump. Why this is needed?. I really dislike AST to support things like AstKeyMaps, but if GaussClump is already using stuff from FITS/WCS, we should consider AST seriously. Comments?

Change the repository name from ACALIB to acalib

What do you think?

UPI documentation

The UPI should have its own and very detailed sphinx documentation, because is the entrypoint for the astronomer.

Integrate Dendograms as an Algorithm

Explore CUPID data structures

CUPID does not only return Clump Assignment Arrays, but there are other several data structures that would be very useful to return to the user. This might need some cython support in the pyCupid project.

Core documentation

The core subpackage does not need to be documented in sphinx, but all functions must include a brief description. Currently half of them have it.

Remove the refactoring branch

We need to remove this branch.

Licensing Code

We need to protect our code (requirement from our funding agency), so every file need to have the GPL license in the header, plus the GPL files required in the main directory.

An example of such header can be found here

scale and get_flux uses python math and not numpy functions

performing a python mathematical operation over a numpy array is terribly SLOW:

In [1]: import timeit
In [2]: import numpy as np
In [3]: a=np.random.random((10,100))
In [4]: f1=lambda :a.sum()
In [5]: f2=lambda : sum(sum(a))
In [6]: %timeit f1()
100000 loops, best of 3: 2.5 µs per loop
In [7]: %timeit f2()
10000 loops, best of 3: 33.4 µs per loop

So... please use numpy!

Change first release from v1.0 to v0.1

I think we need a more mature library for a 1.0 release.

• Change Project
• Update Setup.py / pip.

Noise Level definition

The UPI includes a noise_level() function that is a NDData wrapper of core's rms() function. Here we should check for background estimation rather that simply computing the rms. We might offer the rms as one method of the background estimation (maybe the default).

3D Visualization of a NDData

Provide a Jupyter-friendy visualization of a NDData in 3D (interactive)

Astropythonize the core or not

In my latest refactoring (yep... I do one each month it seems), I've removed the nddata and table support from the core. This was done because the core should be used by the UPI and Algorithms, so we can put all the Astropy data structures (Table and NDData) in these other packages rather than in the core.
I have not done this for all the functions. Therefore,

• We must decide if we will follow this idea
• If yes, then things like example measure_shape(), that returns a astropy table, should we deferred to the UPI and Algorithm subpackages

Integrate the DISPLAY code

The https://github.com/ChileanVirtualObservatory/display repository include code for automatic detection of spectral lines using training data (ML). We need to integrate this to ACALib

Speed up Stacking algorithm

This algorithm works, but we believe it can be largely improved in terms of computational efficiency.

Integrate the SpelAR code

The https://github.com/ChileanVirtualObservatory/SpeLAR repository includes an algorithm for labeling spectral lines using association rules. We need to integrate this code into ACALib.

acalib devel for dummies

Need the commands and procedures to install acalib devel in a conda enviroment. The idea is that a script can do everything so it must be clear and automatic.

Generalize code to support different axes orders

CASA-generated files include stokes and we drop it simply by position. We must free ACALib from hardcoded dimensions (i.e., arr[0] is freq), because that depends on the FITS metada. We need to clarify this, and make sure that we can use ACALib on "any" fits, not only CASA-generated ones.

Acalib import error

I installed acalib with the following commands:

 su - $USER -c "git clone https://github.com/ChileanVirtualObservatory/acalib.git"
 su - $USER -c "cd /home/$USER/acalib && git checkout devel && /home/$USER/conda/bin/pip  install -e ."

and I get this error in a jupyterhub-notebook when import:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-72aa9cf42e28> in <module>()
----> 1 import acalib

~/acalib/acalib/__init__.py in <module>()
      1 from __future__ import absolute_import
      2 
----> 3 from .algorithms import *
      4 from .io import *
      5 from .upi import *

~/acalib/acalib/algorithms/__init__.py in <module>()
      2 from .clumpfind import ClumpFind
      3 from .gms import GMS
----> 4 from .roise import RoiSE
      5 from .stacking import Stacking
      6 from .represent import HRep

ImportError: cannot import name 'RoiSE'

WCS turn around

The WCS pix2world conversion on astropy CORRECTLY turn around the clock when you arrive to a zero limit. For example, if the object is in alpha=0.0 center, then all the pixels to the right will be like delta_something, while pixels to the left will be 365 - delta_something, rather than simply the negative number. This is a complex issue that we must address here!

Verify and test stacking code

The code done by @vansimonsen needs to be reviewed and corrected if needed. Also a notebook with a concrete example should be provided (in the notebooks repository). This is classified as a bug because we already found some problems with the alignments of the images.

Segfault when using Clumpfind repeatedly

The clumFind algorithm continue to crash in some (apparently random) cases. We need to debug this issue.

Python2 errors in Clumping Algorithms

Getting errors when running Clumping Algorithms acalib.Fellwalker, acalib.ClumpFind in python2, the error comes from pycupid (ChileanVirtualObservatory/pycupid#4)

SAMP Support

Using http://astrojs.github.io/sampjs/ we should be able to send NDData and Astropy tables from local apps or from a server (e.g. JupyterHub) to the SAMP Hub.

Enhance description of `acalib.core.transform.fits_props`

"""
Extracts properties information of the astronomical data cube.
Parameters
----------
img : numpy.ndarray
    Astronomical data cube.

Returns
-------
result : dict
    Dictionary with properties of the image: *centroid*, *major*, *minor*, *ratio*, *angle*, *area*, *img*, *clr*, *label*, *orig*.
"""

Doens't seem to be enough for this function that does a lot of things. Maybe describe each value of the output dictionary?

Stacking as an Algorithm

There is no algorithms/stacking.py file, and no inherited class from Algorithm that implement what is already supposed to be working in the core. Remember to document this class

Find a place for jovial stylers

I've developed a few jovial stylers for arrays and tuples with quantities. These are at io/__init__.py right now. We should find a place for this, and decide if goes into the master or the devel branch

Animation of Cubes

The 3D visualization is out of the question for ACALib 1.0. But in the devel branch is commented a code for performing an animation of cubes by frequency. We must have a slider version (similar to CASA or dendroviewer) to visualize a cube.

Non-deterministic error on tests.

run_test.py throws the following error sometimes:

======================================================================
FAIL: test_morph (core.test_analysis.TestAnalysis)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/apawn/Documents/repos/acalib/testing/core/test_analysis.py", line 72, in test_morph
    np.testing.assert_almost_equal(boxing,result_boxing)
  File "/home/apawn/Music/miniconda3/envs/acalibenv/lib/python3.6/site-packages/numpy/testing/utils.py", line 563, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/home/apawn/Music/miniconda3/envs/acalibenv/lib/python3.6/site-packages/numpy/testing/utils.py", line 962, in assert_array_almost_equal
    precision=decimal)
  File "/home/apawn/Music/miniconda3/envs/acalibenv/lib/python3.6/site-packages/numpy/testing/utils.py", line 778, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 7 decimals

(mismatch 10.0%)
 x: array([  6.9413918e-310,   1.0000000e+000,   1.0000000e+000,
         1.0000000e+000,   1.0000000e+000,   1.0000000e+000,
         1.0000000e+000,   1.0000000e+000,   1.0000000e+000,
         8.4286332e+174])
 y: array([  6.9247206e-310,   1.0000000e+000,   1.0000000e+000,
         1.0000000e+000,   1.0000000e+000,   1.0000000e+000,
         1.0000000e+000,   1.0000000e+000,   1.0000000e+000,
         6.9246513e-310])

ClumpFind and FellWalker documentation

There is no documentation of ClumpFind and FellWalker algorithm classes (Sphinx). It will be easier to refer the user to the cupid official documentation rather than explaining the parameters ourselves. The documentation of GaussClump is deferred to the devel branch.

Container.save_fits

It create the fits with a duplicated image (container.primary and first image of container.image array)

APLpyfy the 2D visualization

Currently we are using wcsaxes package for 2D visualization. We can do better by using APLpy. An example can be found here https://github.com/ChileanVirtualObservatory/notebooks/blob/master/course/II%20-%20Accesing%20Data.ipynb

Prepare ACALIB's setup

Prepare system for installing ACALIB:

• Create setup.py
• Check setup to support PIP standards.

Run Tests in Travis Script

The script is just for building the wheel for now, we need to install the wheel and run the tests.

Removing the Container class

The container is meant to read a fits in a generic way, yet I guess this is impossible, because there are too many variations of them. However, what we can do is to provide only the HDU_to_fits() function. This way, the fits reading is responsibility of the user (there are a lot of tutorials!), and we only do the transformation to NDData. What do you think?

Use the NDDataRef interface of astropy

We are currently using the NDData interface for creating and loading data. This should be ported to the new NDDataRef interface, including algorithms, loading and saving using the standard I/O astropy interface and implementing the slicing methods.

Indexing Algorithm Name and Documentation

The name of the class "Indexing" is too general. We should change the name to something more clear, and refer in the documentation to the paper and to the original code. Probably we should discuss the name with @marcelomendoza

Use pyCupid package

We have defined a new repo:

https://github.com/ChileanVirtualObservatory/pycupid

where pyCupid is now.

We need to:

• put this package as dependency of ACALib
• remove from the code the old cupid directory
• change the wrapper functions of FellWalker and ClumpFind at least for using the library

Spectra should be computed from any pixel set

We should use a pixelset structure. The functions like spectra works for a bounding box, which is correct, but we need to support a "pixel set" also.