astropy/ccdproc

Astropy affiliated package for reducing optical/IR CCD data

ccdproc

Ccdproc is is an affiliated package for the AstroPy package for basic data reductions of CCD images. The ccdproc package provides many of the necessary tools for processing of ccd images built on a framework to provide error propagation and bad pixel tracking throughout the reduction process.

Ccdproc can currently be installed via pip or from the source code. For installation instructions, see the online documentation or docs/install.rst in this source distribution.

Documentation is at ccdproc.readthedocs.io

An extensive tutorial is currently in development.

Contributing

We have had the first stable release, but there is still plenty to do!

Please open a new issue or new pull request for bugs, feedback, or new features you would like to see. If there is an issue you would like to work on, please leave a comment and we will be happy to assist. New contributions and contributors are very welcome!

New to github or open source projects? If you are unsure about where to start or haven't used github before, please feel free to email @crawfordsm, @mwcraig or @mseifert. We will more than happily help you make your first contribution.

Feedback and feature requests? Is there something missing you would like to see? Please open an issue or send an email to @mwcraig, @crawfordsm or @mseifert. Questions can also be opened on stackoverflow, twitter, or the astropy email list.

Ccdproc follows the Astropy Code of Conduct and strives to provide a welcoming community to all of our users and contributors.

Want more information about how to make a contribution? Take a look at the astropy contributing and developer documentation.

If you are interested in finacially supporting the project, please consider donating to NumFOCUS that provides financial management for the Astropy Project.

Acknowledgements

If you have found ccdproc useful to your research, please considering adding a citation to ccdproc contributors; Craig, M. W.; Crawford, S. M.; Deil, Christoph; Gasdia, Forrest; Gomez, Carlos; Günther, Hans Moritz; Heidt, Nathan; Horton, Anthony; Karr, Jennifer; Nelson, Stefan; Ninan, Joe Phillip; Pattnaik, Punyaslok; Rol, Evert; Schoenell, William; Seifert, Michael; Singh, Sourav; Sipocz, Brigitta; Stotts, Connor; Streicher, Ole; Tollerud, Erik; and Walker, Nathan, 2015, Astrophysics Source Code Library, 1510.007, DOI: 10.5281/zenodo.47652

Thanks to Kyle Barbary (@kbarbary) for designing the ccdproc logo.

ccdproc's Issues

Fix modeling calls to match API in astropy 0.3

Right now the test test_subtract_overscan_model fails on the call to models.polynomial.Poly1DModel

Add cosmicray_lapace

Add cosmic ray cleaning following LA-cosmic:
http://arxiv.org/abs/astro-ph/0108003

Change instances of header.set(key, value) to header[key] = value in tests

subtract_overscan doesn't correctly test ccd type

Implement non-linear gain correct

Add test of cosmicray_clean gbox

Not clear to me what the desired outcome is here -- @crawfordsm can you explain?

extract unit from image header

If units are specified in the image header, the fits_ccddata_reader should be able to extract the units from the header and they should not need to be explicitly stated in the header.

Get to 100% test coverage for modules that are specific to this affiliated package

In other words, ccdproc.ccdproc and cdproc.ccddata for now. The other code that isn't covered largely comes from the astropy affiliate template.

ccdproc.transform

ccdproc tasks for transforming data. This should be able to handle general transformations such as shift, rotation, and magnification.

logging with no add_keywords

Logging does not seem to behave as expected in the api. In the api, we suggest that the if add_keyword is not specified, the task name and parameters should be added to meta, but that does not seem like the case right now.

cosmicray_clean changes images in-place instead returning a copy

Set up coveralls.io for test coverage badge

Make PEP8 fixes

These appear to be mostly minor whitespace fixes (e.g. spaces around = and after ,) but there may be a couple of camelCaseNames that should be converted to names_with_underscores

Drop requirement that metedata be stored internally as a fits header

create_variance makes an uncertainty which is a Quantity

This is a problem because NDDData doesn't function correctly with Quantity arrays. One of the issues this creates is that the unit of the uncertainty can be completely different (and inconsistent with) the unit of the CCDData.data.

sigma_clip should allow clipping along stack only

In some cases (e.g. creating a master dark) there pixels which are, in any individual image or the set of images being combined, several standard deviations from the mean or median. In the case of combining to make a master dark these are hot pixels, and should not be masked in sigma_clip.

The clipping should exclude those pixels which vary by more than a few standard deviations from the mean of that pixel over all images.

Make test_flat_correct check for something

Right now it creates data and flat images then just does:

    ccd = flat_correct(ccd, flat)

ccdproc.distortion_correct

Apply distortion corrections by either updating the WCS or transform the frame. Basic functionality will probably be included in ccdproc.transform and more advance WCS handling will require the WCS class.

synonyms in Keyword

synonyms has not been implement in keyword.

minvalue in flat_correct

minvalue in flat_correct has not been implemented

Use astropy.units definitions of adu, electrons, etc where possible

In the time since ccdproc was originally written the some of the units defined in ccdproc.ccddata have been defined in astropy.units

rejection algorithms

Should the clipping algorithms be moved out of combiner and into there own class/functions ?

checking size of array before trimming

In ccdproc.trim_image, there is no checking to see if the requested section is within the boundaries of the image. This seems to be the same behaviour as ndarray.

Should we keep this behavior or add some check to make sure that the trimmed area is within the give ccddata object?

Propagate errors when doing scalar op CCDData

Should be able to handle multiplication, division, addition, subtraction of a scalar without an error and a CCDData object, and pass through to the NDData methods .add etc if the other operand is not a numbers.Number.

cosmicray_clean ignores ``background`` if it is float or ndarray

Relevant code snippet is below. Handles None or a function only.

    if background is None:
        background = sigma_func(data)
    elif hasattr(background, '__call__'):
        background = background(data, *bargs)

Fitting models with rejection

Fitting models with rejecting points is a need for the ccdproc package, for example in fitting the overscan region. However, this is probably something that astropy.modeling will eventually take care of, but in the meantime, we should supply something.

weighting in combiner

Add calculating the weighting to combiner or provide other functions for doing it

Add check to CCDData.create_variance for units of readnoise

Right now the code checks whether the CCDData object has units of electrons, but not whether the readnoise is also in electrons

Several of the functions in ccdproc.ccdproc do not actually return CCDData objects

Not sure this is the end of the world, but they tend to return whatever type was passed in (e.g. a numpy array).

I'd lean towards ensuring that a CCDData object is returned.

ccdproc.combine

Convenience function and wrapper for combiner

subtract_oscan assumes overscan is always the second axis

e.g.:

oscan = np.median(overscan.data, axis=1)

Working on a fix...

Start setting up documentation...

...with at least the correct package name and auto-generated API docs

fringe correction

fringe correction is likely a convenience function (as it is either a flat_image or a scaled subtraction) and so not necessary to implement right away. It is also something that will be telescope dependent.

fits_section doesn't do any bounds checking

At a minimum it should check for a zero index on input and refuse to decrease it to -1. Perhaps it should also check for negative index as the stop and disallow that too.

Set up travis builds

Should test at least python 2.7, 3.3.

ccdproc.rebin

Function for rebinning ccddata data. It should handle rebinning mask and uncertainty frames as well.

Decide whether functions like subtract_bias allow CCDData or nddata as arguments

The API and docstring imply CCDData only (which seems like the way to go) but the type check is currently isinstance(ccd, CCDData) or isinstance(ccd, np.ndarray)

Change unit names to singular to match astropy convention

This boils down to changing electrons to electron, I think...

Find out how to register a fits reader/writer in python 3....

Travis builds on #35 are failing in python 3 because the fits reader and the meta in ccddata use iteritems, which went away in python 3. six provides a workaround.

Writing out mask, flags, uncertainty information

For now any flag, mask and/or unit for ccddata is discarded when writing. If you want all or some of that information preserved you must create the FITS files manually.

It would be good to be able to write out that information as well.

Implement cross-talk correction

background in cosmicray_median

Docs should be updated and an error test should be added that background in cosmicray_median should be specified

scaling in combiner

Implement some scaling in combiner

Set up ReadTheDocs

Fix sphinx build warnings

These are generated because automodapi tries to generate docs for things we import -- one fix is to set __all__ to something sensible in each module.

...
/Users/mcraig/Development/astronomy/ccdproc/docs/ccdproc/index.rst:49: WARNING: toctree references unknown document u'api/ccdproc.ccdproc.Quantity'
/Users/mcraig/Development/astronomy/ccdproc/docs/ccdproc/index.rst:49: WARNING: toctree references unknown document u'api/ccdproc.ccdproc.StdDevUncertainty'
...
None:None: WARNING: toctree contains reference to nonexisting document u'api/ccdproc.ccdproc.Quantity'
None:None: WARNING: toctree contains reference to nonexisting document u'api/ccdproc.ccdproc.StdDevUncertainty'
...

basic arithmetic

Having things like

ccd = cccddata + 2 
ccd = ccddata + ccddata

working. At the current time, basic arithmetic with CCDData can be done using .add for example. However, implementing the above, if ever, will require changes to NDData and Quantity.

subtract_dark should be able to scale the dark frame by exposure time

It is in the API but right now can't be done....

Add a basic example to the docs

Could you please add a basic example to the docs to show what ccdproc is about and what it can currently do?

I mean something short like the "Introduction" and "Getting Started" sections in photutils.

masked median_combine is excruciatingly slow

This is a known issue, though with no solution in numpy: numpy/numpy#4683

By "excruciating" I mean masked median is roughly 1,000x slower than unmasked median.

Summary of several options (the "reference" time is np.median with no masks):

function	masked?	time (sec)	slowness factor
np.median	no	1.07s	1x
np.ma.median	yes	838s	783x
loop over pixels	yes	254s	237x
scipy.stats.nanmedian	yes	232s	216x
bottleneck.nanmedian	yes	1.35s	1.26x

notebook with timings is at: http://nbviewer.ipython.org/gist/mwcraig/9711663484d7cc264fa7

The fastest by far is bottleneck but requiring bottleneck adds a dependency.

One way around this is:

def median_combine(self, median_nan_func=None):
"""
...
           Parameters
           ----------
           median_nan_func : function, optional
               Function that calculates median of a ``numpy.ndarray``, ignoring any
               values that are ``NaN``. Default is to use ``np.ma.median`` to
               calculate median.
"""

Standardize handling of tests in which exceptions are expected

RIght now some of the tests use the usual (for pytest) way with pytest.raises(...) and some use a decorator @raises(...)