lgrcia/prose

Modular image processing pipelines with Python. Built for Astronomy.

prose

Modular image processing pipelines for Astronomy

prose is a Python package to build modular image processing pipelines for Astronomy.

powered by astropy and photutils!

Example

Here is a quick example pipeline to characterize the point-spread-function (PSF) of an example image

import matplotlib.pyplot as plt
from prose import Sequence, blocks
from prose.simulations import example_image

# getting the example image
image = example_image()

sequence = Sequence(
    [
        blocks.PointSourceDetection(),  # stars detection
        blocks.Cutouts(shape=21),  # cutouts extraction
        blocks.MedianEPSF(),  # PSF building
        blocks.Moffat2D(),  # PSF modeling
    ]
)

sequence.run(image)

# plotting
image.show()  # detected stars

# effective PSF parameters
image.epsf.params

While being run on a single image, a Sequence is designed to be run on list of images (paths) and provides the architecture to build powerful pipelines. For more details check Quickstart and What is a pipeline?

Installation

latest

prose is written for python 3 and can be installed from pypi with:

pip install prose

For the latest version

pip install 'prose @ git+https://github.com/lgrcia/prose'

Contributions

See our contributions guidelines

Attribution

If you find prose useful for your research, cite Garcia et. al 2022. The BibTeX entry for the paper is:

@ARTICLE{prose,
       author = {{Garcia}, Lionel J. and {Timmermans}, Mathilde and {Pozuelos}, Francisco J. and {Ducrot}, Elsa and {Gillon}, Micha{\"e}l and {Delrez}, Laetitia and {Wells}, Robert D. and {Jehin}, Emmanu{\"e}l},
        title = "{PROSE: a PYTHON framework for modular astronomical images processing}",
      journal = {\mnras},
     keywords = {instrumentation: detectors, methods: data analysis, planetary systems, Astrophysics - Instrumentation and Methods for Astrophysics, Astrophysics - Earth and Planetary Astrophysics},
         year = 2022,
        month = feb,
       volume = {509},
       number = {4},
        pages = {4817-4828},
          doi = {10.1093/mnras/stab3113},
archivePrefix = {arXiv},
       eprint = {2111.02814},
 primaryClass = {astro-ph.IM},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2022MNRAS.509.4817G},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

and read about how to cite the dependencies of your sequences here.

prose's Issues

An html notebook widget to explore apertures

Could think about a richer view, like zoomable stack

Case studies

Case studies including:

• Multi extensions fits
• Fully custom apertures

prose 3.0.0 : problem installation on Windows (jaxlib)

It seems that Jaxlib (required to install prose 3.0.0) is not supported by Windows ( cfr https://stackoverflow.com/questions/62585395/not-able-to-install-jaxlib )

Possible solution: https://github.com/cloudhan/jax-windows-builder

Available telescopes not updated when adding new

allow to exclude comparison stars from Broeg

It would be nice to be able to exclude certain comparison stars, either because they look blended or because they are higly variable etc.

Somthing like:

df = obs.broeg2005(exclude_comps = [1,56,86])

More tests and switch from unittest to pytest

Develop more systematic testings (per block and global) and switch the testing framework to pytest

Plate solving tutorial error

When I run Plate solving tutorial follow document, I got an error.
Here is my code:

from prose import Sequence, blocks

plate = Sequence([
    blocks.detection.SegmentedPeaks(min_separation=15, n_stars=15),
    blocks.catalogs.PlateSolve(debug=True)
])

plate.run(image, show_progress=False)

and the error:

Output exceeds the [size limit](command:workbench.action.openSettings?%5B%22notebook.output.textLineLimit%22%5D). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?fe124561-7179-4dfa-8328-049ad16dff0c)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[e:\03_Projects\ImageProcess\imlt\prose\PlatSolving.ipynb](file:///E:/03_Projects/ImageProcess/imlt/prose/PlatSolving.ipynb) Cell 14 in ()
      [1](vscode-notebook-cell:/e%3A/03_Projects/ImageProcess/imlt/prose/PlatSolving.ipynb#X16sZmlsZQ%3D%3D?line=0) from prose import Sequence, blocks
      [3](vscode-notebook-cell:/e%3A/03_Projects/ImageProcess/imlt/prose/PlatSolving.ipynb#X16sZmlsZQ%3D%3D?line=2) plate = Sequence([
      [4](vscode-notebook-cell:/e%3A/03_Projects/ImageProcess/imlt/prose/PlatSolving.ipynb#X16sZmlsZQ%3D%3D?line=3)     blocks.detection.SegmentedPeaks(min_separation=15, n_stars=15),
      [5](vscode-notebook-cell:/e%3A/03_Projects/ImageProcess/imlt/prose/PlatSolving.ipynb#X16sZmlsZQ%3D%3D?line=4)     blocks.catalogs.PlateSolve(debug=True)
      [6](vscode-notebook-cell:/e%3A/03_Projects/ImageProcess/imlt/prose/PlatSolving.ipynb#X16sZmlsZQ%3D%3D?line=5) ])
----> [8](vscode-notebook-cell:/e%3A/03_Projects/ImageProcess/imlt/prose/PlatSolving.ipynb#X16sZmlsZQ%3D%3D?line=7) plate.run(image, show_progress=False)

File [c:\Users\Alfre\AppData\Local\Programs\Python\Python310\lib\site-packages\prose\core\sequence.py:103](file:///C:/Users/Alfre/AppData/Local/Programs/Python/Python310/lib/site-packages/prose/core/sequence.py:103), in Sequence.run(self, images, terminate, show_progress, loader)
    101 self.n_processed_images = 0
    102 self.discards = {}
--> 103 self._run(loader=loader)
    105 if terminate:
    106     self.terminate()

File [c:\Users\Alfre\AppData\Local\Programs\Python\Python310\lib\site-packages\prose\core\sequence.py:119](file:///C:/Users/Alfre/AppData/Local/Programs/Python/Python310/lib/site-packages/prose/core/sequence.py:119), in Sequence._run(self, loader)
    116 image.i = i
    118 for block in self.blocks:
--> 119     block._run(image)
    120     # This allows to discard image in any Block
    121     if image.discard:

File [c:\Users\Alfre\AppData\Local\Programs\Python\Python310\lib\site-packages\prose\core\block.py:54](file:///C:/Users/Alfre/AppData/Local/Programs/Python/Python310/lib/site-packages/prose/core/block.py:54), in Block._run(self, *args, **kwargs)
...
File [c:\Users\Alfre\AppData\Local\Programs\Python\Python310\lib\site-packages\prose\blocks\registration.py:10](file:///C:/Users/Alfre/AppData/Local/Programs/Python/Python310/lib/site-packages/prose/blocks/registration.py:10), in distance(p1, p2)
      9 def distance(p1, p2):
---> 10     return np.sqrt(np.power(p1[0] - p2[0], 2) + np.power(p1[1] - p2[1], 2))

TypeError: 'PointSource' object is not subscriptable

My system is Win11 22624.1391, Python version is 3.10.9, prose version is 2.3.0.

Bug loading phot when folder has telescope name

When loading telescope (e.g. loading a .phot), error when a root directory has same name has telescope ('cause Telescope.load first check a file with telescope name in current folder in case of a custom telescope file)

Automatic release

Setup an automatic release pipeline for Gihtub and pypi

Account for saturated stars in keep_good_stars

Add import_stack to documentation (useful tips)

Typo in to_csv() function

In prose/observation.py line 89, it is written:

"ERROR": self.error

But this creates an eror as it should be:

"ERROR": self.diff_error

`'` in paths breaks `FitsManager`

' character in absoulte paths of files breaks some of the behaviours of prose.FitsManager

Display observation id in FitsManager calib table

Bad values in stack detection [TS 20210728]

Bad values in stack detection (night 2021-07-28 TS)

Add sum of comparison stars in `Observation.plot_comp_lcs()`

Guidelines on plate solving

Write guidelines on the parameters to use when doing plate solving.

There is a trade-off between the number of stars to consider against the number of queried stars to match (more stars make the match slower). For example, we retain the n brightest Gaia stars in a field s times larger than the expected field of view (using image size and pixel known plate scale), and we compare these stars to the m brightest detected stars in the image. All these parameters play an important role, especially since brightness of Gaia stars is sorted from phot_g_mean_fluxes that may not match the relative brightness of the stars in a given observation with different wavelength sensitivity.

Some guidelines would be useful (see #58)

Add fixed transit depth for systematics model comparison

An image discarded in AffineTransform makes xarray alignment fail

In pipeline.Calibration, if an image is discarded by blocks.AffineTransform it is not retained in the final blocks.Xarray but is still saved reduced (blocks.SaveReduced appears before). Hence the pipeline.Photometry will fail (taking from the calibration xarray which has less images than the number reduced ones).

Possible solutions:

• Realign xarray to times
• produce an independent xarray in Photometry
• reorganise the Calibration...

Fwhm computation inverted in plot_psf_model and _compute_psf_model

Using obs._compute_psf_model() or obs.plot_psf_model() gives inverted values of obs.stack.fwhmx and obs.stack.fwhmy.

Mention that fm contains reduced

A more practical bjd conversion and time format managment

Specify `date` in Telescope definition

Add a date to the telescope definition so that definiton is date dependant:

Telescope({...}, date="2018-12-01")

For example if an image is taken on the 2018-12-13 and the following telescopes are available:

2018-11-12_name.telescope
2018-12-01_name.telscope
2022-10-01_name.telescope

2018-12-01_name.telscope is chosen, as the latest past telescope w.r.t image date

Allow `dates` and `between` in FitsManager methods like describe

typo: missing .stack in Observation.compute_bjd()

Typo in version= "eastman in Observation.compute_bjd(), lines 301-302

current code:

elif version == "eastman":
            bjd_time = utils.jd_to_bjd(self.jd_utc + exposure_days/2, self.stack.skycoord.ra.deg, self.skycoord.dec.deg)

should be:

elif version == "eastman":
            bjd_time = utils.jd_to_bjd(self.jd_utc + exposure_days/2, self.stack.skycoord.ra.deg, self.stack.skycoord.dec.deg)

Reimplement better dependencies citation system

To cite packages used in blocks and sequences in a correct way.

Handle failing twirl in Calibration

Handle failing twirl in calibration

Error we got is in blocks/registration.py line 354
i, j = tutils.cross_match(new_ref, s, tolerance=tolerance, return_ixds=True).T
got
ValueError: not enough values to unpack (expected 2, got 0)

Global light curves

Defocused PSF model Block

Reduction does not work without Calibrations

Even if i set calibration = False reduction is failling if calibrations images are missing.

add blocks running time view in doc (extra)

How to put a star out of consideration of Broeg2005()?

In some cases, the Broeg2005() gives "two" comparison stars as the best ones for differential photometry. One of the two stars presents some variations exactly during the transit time, which may have an impact one the transit signal. Is there any option to put that star out of Broeg2005() consideration?
Thanks in advance ;)

Add quick TIC url

function to generate url to exoFOP page of a tic

Special characters in telescope names

When the TELESCOP keyword in the FITS header contains a special character, eg: EQMOD ASCOM HEQ5/6, prose fails. The workaround I have found for this is editing the TELESCOP field to another field (in my case SITENAME) in following lines:

Line 83 in IO.PY - telescope_kw="SITENAME"
Line 118 in IMAGE.PY - self.header.get("SITENAME", "")

It would be good if prose could pull the keyword designed for keyword_telescope: SITENAME or name: from the .telescope files

Thanks a lot
Alex

Add centroiding option for Observation.plot_psf

error in reduction.run()

I got the following error during the reduction :

RUN: Reduction: 18%|███▊ | 199/1096 [01:27<06:24, 2.33files/s]WARNING: File may have been truncated: actual file length (2018364) is smaller than the expected size (8562240) [astropy.io.fits.file]
TypeError: buffer is too small for requested array

when I open the corresponding file I get the following error in ds9: a SIGBUS or SIGSEGV error has been received

It looks like some files are corrupted.

stack.fits is reference image and not stack image

In prose/pipeline/calibration.py, the file saved to stack_path is self.reference and not self.stack. The FitsManager was not recognizing the stack image because it said "Light frame" in the header.

Add RA and DEC for selected star in summary plot

I think it would be a nice idea if there would be a way to see the RA and DEC for the selected star in the summary plots. At the moment the RA and DEC on the summary plots are those from the centre of the image.

I see the following block of code (lines 1011-1018) in observation.py

params = [ ("Telescope", f"{image.telescope.name}"), ("Date", f"{image.night_date}"), ("Filter", f"{image.filter}"), ("Exposure", f"{image.exposure}"), ("RA", f"{image.skycoord.ra:.4f}"), ("DEC", f"{image.skycoord.dec:.4f}"), ("Dimenion", f"{image.shape[0]}x{image.shape[1]} pixels"),
]

I think it would be good to have the option to change the RA and DEC line from skycoord.ra and skycoord.dec to something like stars_coords.ra and stars_coords.dec

Many thanks,
Alex

``_stars_safe`` attribute in ``Image``

An attribute in classes that turns False (unsafe) when a set of stars coordinates has been used by a block (for example to extract photometry) but is modified by another block.

A possible behaviour would be:

s = Sequence([
    blocks.StarsDetection(),
    blocks.Photometry(),
    blocks.LimitStars(min=5)
], stars_safe=True) # True by default

Warning: stars_coords have been modified after beeing used in block `Photometry`

Through a Sequence._stars_safe updated after every Image is ran

Detection on stack failed [20180826 SSO/Io]

Probably no stars found on stack:

File ".../prose/prose/pipeline/photometry.py", line 94, in run
self.detection_s.run(show_progress=False)
...
File ".../prose/prose/blocks/detection.py", line 91, in single_detection
coordinates = np.transpose(np.array([sources["xcentroid"].data, sources["ycentroid"].data]))
TypeError: 'NoneType' object is not subscriptable

example on 20180826 SSO/Io

Create built ins telescopes files

Bad ref image [20180806 SSO/Io]

Deal with the case where reference image in Calibration is bad, e.g. leading to:
Warning: image 0 discarded in SegmentedPeaks

Specify git branch name and commit hash in `phot` file

typo in default file for Liverpool Telescope

The longitude of the site add an extra minus sign which was creating some errors in the computation of BJT_TDB times.

Must change in /.prose/liverpool.telescope:

latlong:
- 28.7624
- -17.8792

into:

latlong:
- 28.7624
- 17.8792

Set image plots axis with physical limits

Set image plots axis with physical limits like RA and DEC range, or arc-seconds. Useful to compare images of different resolutions

add PANSTARRS finderchart

Add PANSTARS image in findercharts (e.g. https://ps1images.stsci.edu/cgi-bin/ps1cutouts?pos=--&filter=z&filetypes=stack&auxiliary=data&size=1200&output_size=0&verbose=0&autoscale=99.500000&catlist=)

Optimize segmented peaks

Put a limit on filled region areas

missing doc

function .save_mcmc_file is not documented

Add prose version in .phot file

Change how PSF fitting blocks work

Add an EPSF Block and a espf kwarg in the PSF fitting blocks so that they are applied on the epsf, otherwise on all cutouts stored in blocks.cutouts