bwinkel/cygrid

Cygrid is a cython-powered convolution-based gridding module for astronomy

cygrid

• Version: 2.0
• Authors: Benjamin Winkel, Lars Flöer, Daniel Lenz
• User manual: stable | developer

Project Status

Cygrid is already used in several "production" systems, for example it was utilized for two major 21-cm HI surveys, EBHIS and HI4PI. Nevertheless, we cannot guarantee that it's completely bug-free. We kindly invite you to use the library and we are grateful for feedback. Note, that work on the documentation is still ongoing.

Purpose

cygrid allows to resample a number of spectra (or data points) to a regular grid - a data cube - using any valid astronomical FITS/WCS projection (see http://docs.astropy.org/en/stable/wcs/).

The method is a based on serialized convolution with finite gridding kernels. Currently, only Gaussian (radial-symmetric or elliptical) kernels are provided (which has the drawback of slight degradation of the effective resolution). The algorithm has very small memory footprint, allows easy parallelization, and is very fast.

A detailed description of the algorithm is given in Winkel, Lenz & Flöer (2016), which we kindly ask to be used as reference if you found cygrid useful for your research.

Features

• Supports any WCS projection system as target.
• Conserves flux.
• Low memory footprint.
• Scales very well on multi-processor/core platforms.

Installation

We highly recommend to use cygrid with the Anaconda Python distribution, in which case installiation is as easy as

conda install -c conda-forge cygrid

Otherwise, you should install cygrid via pip:

python -m pip install cygrid

The installation is also possible from source, but you'll need a C++ compiler. Download the tar.gz-file, extract (or clone from GitHub) and execute (in project directory):

python -m pip install .

Dependencies

We kept the dependencies as minimal as possible. The following packages are required:

• Python 3.8 or later (cygrid versions prior to v1.0 support Python 2.7)
• NumPy 1.13 or later
• Cython 3 or later (if you want to build cygrid yourself)
• Astropy 4.0 or later

(Older versions of these libraries may work, but we didn't test this!)

If you want to run the notebooks yourself, you will also need the Jupyter server, matplotlib and wcsaxes packages. To run the tests, you'll need HealPy.

Note, for compiling the C-extension, openmp is used for parallelization and some C++11 language features are necessary. If you use gcc, for example, you need at least version 4.8 otherwise the setup-script will fail. It is highly recommended to use Anaconda, which offers the proper compilers for many platforms.

Usage

Minimal example

Using cygrid is extremely simple. Just define a FITS header (with valid WCS), define gridding kernel and run the grid function:

from astropy.io import fits
import cygrid

# read-in data
glon, glat, signal = get_data(...)

# define target FITS/WCS header
header = {
    'NAXIS': 3,
    'NAXIS1': 101,
    'NAXIS2': 101,
    'NAXIS3': 1024,
    'CTYPE1': 'GLON-SFL',
    'CTYPE2': 'GLAT-SFL',
    'CDELT1': -0.1,
    'CDELT2': 0.1,
    'CRPIX1': 51,
    'CRPIX2': 51,
    'CRVAL1': 12.345,
    'CRVAL2': 3.14,
    }

# prepare gridder
kernelsize_sigma = 0.2

kernel_type = 'gauss1d'
kernel_params = (kernelsize_sigma, )
kernel_support = 3 * kernelsize_sigma
hpx_maxres = kernelsize_sigma / 2

mygridder = cygrid.WcsGrid(header)
mygridder.set_kernel(
    kernel_type,
    kernel_params,
    kernel_support,
    hpx_maxres
    )

# do the gridding
mygridder.grid(glon, glat, signal)

# query result and store to disk
data_cube = mygridder.get_datacube()
fits.writeto(
    'example.fits',
    header=header, data=data_cube
    )

More use-cases and tutorials

Check out the user manual or the Jupyter tutorial notebooks in the repository for further examples of how to use cygrid. Note that you can only view the notebooks on GitHub, if you want to edit something it is necessary to clone the repository or download a notebook to run it on your machine.

Who do I talk to?

If you encounter any problems or have questions, do not hesitate to raise an issue or make a pull request. Moreover, you can contact the devs directly:

• <[email protected]>
• <[email protected]>

Preferred citation method

Please cite our paper if you use cygrid for your projects.

@ARTICLE{2016A&A...591A..12W,
    author = {{Winkel}, B. and {Lenz}, D. and {Fl{\"o}er}, L.},
    title = "{Cygrid: A fast Cython-powered convolution-based gridding module for Python}",
    journal = {\aap},
    archivePrefix = "arXiv",
    eprint = {1604.06667},
    primaryClass = "astro-ph.IM",
    keywords = {methods: numerical, techniques: image processing},
    year = 2016,
    month = jun,
    volume = 591,
    eid = {A12},
    pages = {A12},
    doi = {10.1051/0004-6361/201628475},
    adsurl = {http://adsabs.harvard.edu/abs/2016A%26A...591A..12W},
    adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

cygrid's Issues

anaconda: binary incompatibility on cygrid support

I get a strange error on my existing Anaconda installation, after I installed the new 1.0.1 version of cygrid from conda-forge:

import cygrid                                                           
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-f2e9d062c951> in <module>
----> 1 import cygrid

~/local/miniconda3new/lib/python3.6/site-packages/cygrid/__init__.py in <module>
     21 if not _ASTROPY_SETUP_:
     22     # For egg_info test builds to pass, put package imports here.
---> 23     from .cygrid import *
     24     from .healpix import *
     25     from .hphashtab import *

__init__.pxd in init cygrid.cygrid()

ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject

It may be related to some old compilers/c-libs being present in my site-packages, because it works if I do it in a pristine installation:

conda create -n cygrid-test-3.6 python=3.6 python cygrid --channel conda-forge
activate cygrid-test-3.6
python
import cygrid

(Likewise for Python 3.7)

Let's wait if other users run into the same trouble, before we pursue this issue further.

Better handling of 2D and 3D scenarios

The longer I use cygrid, the more inconvenient it feels that cygrid always needs a third axis, even for continuum data. This often means, one needs to create a copy of the (2D) FITS header, and add a third dimension, before it can be plugged to the WcsGrid constructor. Likewise, the input data array would need an explicit reshape to add the spectral axis, even though it is redundant.

The reasons for this design choice are that the underlying Cython/C++ code cannot easily handle numpy broadcasting rules. Therefore, my guess would be that under the hood it will always be simpler to assume the 3D case. However, one could perhaps change the higher-level handlers to add the 3rd dimension on-the-fly to relieve the burden from the end user.

Don't pin dependencies

I wanted to try out cygrid, but get this error:

$ pip install cygrid --user
Collecting cygrid
  Downloading cygrid-0.9.4.tar.gz (575kB)
    100% |████████████████████████████████| 583kB 755kB/s 
    Complete output from command python setup.py egg_info:
    Package astropy has version 1.2.dev15642
    Version 1.0 required.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/pip-build-fudv73g8/cygrid/

This is because you pin dependencies to exact versions in your setup.py (see here):

dependencies = [['cython', '0.20.2'], ['numpy', '1.8'], ['astropy', '1.0']]

This is bad, it will prevent many users to install and use your package. Maybe they have another code that needs Astropy 1.1 or just don't want to install a separate environment with exactly these versions just to get cygrid.

Instead do something like in the example here and put e.g.

install_requires=[
      'numpy>=1.10',
      'astropy>=1.0',
    ],

The example 01_minimal_example.ipynb works perfectly with one minor addition

With one very minor issue, the minimal example worked perfectly within
the datalab.noirlab.edu environment, with one minor addition.

Just before the "import cython" notebook entry, add this cell:
! pip install cygrid

After a few warnings, this completely installs cygrid
(alternately add a "try" statement around the import cygrid step)

Then each step in the minimal example worked fine and yielded the images in the demo documentation.

Thanks for your great work!
Glen

4D data support

Dear cygrid mantainers,

I'm seeking for an alternative to scipy.interpolate.griddata and I just stumbled upon this package. Currently I am working with somewhat large 3-D data arrays (i.e. as in cartesian x, y, z, value), but I have only seen examples in your documentation dealing with the 2D-scenario (i.e. x, y, value).

I was wondering if cygrid supports such an scenario, and if that were the case if you could point me to how to use the package towards that goal.

Many thanks in advance,

Update PyPI requirements

Currently lists Py>=2.7, update to Py>=3.6

clang error that -fopenmp is unsupported on Mac

I'm getting this error:

$ pip install cygrid --user
Collecting cygrid
  Downloading cygrid-0.9.5.tar.gz (582kB)
    100% |████████████████████████████████| 583kB 1.0MB/s 
Requirement already satisfied (use --upgrade to upgrade): setuptools in /Users/deil/Library/Python/3.5/lib/python/site-packages (from cygrid)
Requirement already satisfied (use --upgrade to upgrade): cython>=0.20.2 in /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (from cygrid)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.8 in /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (from cygrid)
Requirement already satisfied (use --upgrade to upgrade): astropy>=1.0 in /Users/deil/Library/Python/3.5/lib/python/site-packages/astropy-1.2.dev15642-py3.5-macosx-10.11-x86_64.egg (from cygrid)
Installing collected packages: cygrid
  Running setup.py install for cygrid ... error
    Complete output from command /opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/private/var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/pip-build-n2qwoo50/cygrid/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/pip-qqnodoyd-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.11-x86_64-3.5
    creating build/lib.macosx-10.11-x86_64-3.5/cygrid
    copying cygrid/__init__.py -> build/lib.macosx-10.11-x86_64-3.5/cygrid
    running build_ext
    cythoning cygrid/kernels.pyx to cygrid/kernels.cpp
    building 'cygrid.kernels' extension
    creating build/temp.macosx-10.11-x86_64-3.5
    creating build/temp.macosx-10.11-x86_64-3.5/cygrid
    /usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -pipe -Os -I/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/numpy/core/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c cygrid/kernels.cpp -o build/temp.macosx-10.11-x86_64-3.5/cygrid/kernels.o -fopenmp -O3 -std=c++11
    clang: error: unsupported option '-fopenmp'
    error: command '/usr/bin/clang' failed with exit status 1

    ----------------------------------------
Command "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/private/var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/pip-build-n2qwoo50/cygrid/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/pip-qqnodoyd-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /private/var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/pip-build-n2qwoo50/cygrid/

I think most people on Mac will hit this error, i.e. effectively install is currently broken on Mac.

Do you want to support Mac (and Windows)?

If yes, the first step should be to add a Mac build on travis-ci and a Windows build on Appveyor.

I remember some discussion and investigation on how to handle OpenMP compilation here:
astropy/imageutils#35 (comment)

The solution they implemented is here:
https://github.com/astropy/astroscrappy/blob/e28afa4d27201df87044625e0e23147778fb70a1/astroscrappy/utils/setup_package.py#L23

It's pretty complex though and uses astropy-helpers, i.e. is a non-trivial change for you and probably a lot of tedious work to get this working properly. Personally I don't really need this package, I just wanted to try it out, so don't feel the need to go down that path on my behalf.

Maybe for now add a comment to the README that this is a known issue on Mac?

healpix regrid notebook issue

Hi there,

I was trying to open the 03_regrid_from_healpix.ipynb tutorial notebook, but jupyter is complaining:

File Load Error for 03_regrid_from_healpix.ipynb
Unreadable Notebook: /Users/dancpr/Data/install/py/cygrid/notebooks/03_regrid_from_healpix.ipynb NotJSONError('Notebook does not appear to be JSON: \'{\\n "cells": [\\n {\\n "cell_type": "m...')

Not sure why, as all the other notebooks load fine.

FWIW, we are hoping to use cygrid to convert all-sky FITS images from radio data into healpix, e.g. converting this:

to this:

We noticed our simple nearest neighbour method introduced artifacts, so something like cygrid seems the way to go!

ImportError: No module named hprainbow

Hi, just installed this package, and the title says it all:

$ ipython2
Python 2.7.11 (default, Mar 31 2016, 06:18:34)
Type "copyright", "credits" or "license" for more information.

IPython 4.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import cygrid
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-71adc91511bd> in <module>()
----> 1 import cygrid

/usr/lib/python2.7/site-packages/cygrid/__init__.py in <module>()
      1 from .cygrid import *
      2 from .healpix import *
----> 3 from .hprainbow import *
      4 from .helpers import *
      5

ImportError: No module named hprainbow

I assume the relevant file is not on the public git repo?

Update dependencies in README

Dependencies are not up to date, we should take a look at this. Also link to INSTALL

gcc 4.7+ needed

Thanks for making this package available, looking forward to trying it.

Just a quick note regarding the installation: Both installation via pip and compilation from source fail when using an older gcc version than 4.7. This is simply because for gcc < 4.7 the flag for C++11 is "-std=c++0x" and not "-std=c++11" (e.g. http://stackoverflow.com/questions/14674597/cc1plus-error-unrecognized-command-line-option-std-c11-with-g). This is not very hard to track down, but maybe a note can be added to the installation instructions. Although gcc 4.7 has been around for quite a while, some systems (such as mine) still seem to have older versions as the default.

How does this compare to the reproject package?

So far I've been using the https://github.com/astrofrog/reproject package?

How does this package compare in terms of features and speed?
I think the algorithm is different, right?

I'll print the paper now and have a look soon:
http://www.aanda.org/articles/aa/full_html/2016/07/aa28475-16/aa28475-16.html

Feel free to close this issue if you don't want to investigate / discuss this.
But if you do ... as a new user that found your package more of less by chance ... I would have found a link to the paper and a short description how cygrid compares to reproject useful.