cylammarco/aspired

Automated SpectroPhotometric Image REDuction (ASPIRED)

Hey 👋, I'm Marco (@cylammarco)

I'm a senior researcher/software developer at The University of Edinburgh, working with the Euclid consortium. My researches cover the topic of white dwarfs⭐, faint blue pulsators🌟, star formation history of the Galaxy🌌, and spectral data reduction software development 🏳️‍🌈👾.

• 🤗 Personal website
• 🐣 Twitter
• 🧻 Google Scholar
• 🔭 I’m currently maintaining ASPIRED, RASCAL, WDPhotTools & SpectResC

For the reason of work, I have been using a private GitLab host since late 2023. I am still actively coding :)

aspired's Issues

Refactor ap_extract to allow more efficient calling and future extension

Marsh89 algorithm depends on Horne86 algorithm, which in turn requires tophat (standard) extraction. These extraction methods should be turned into internal (nested) functions, while the ap_extract() handles the choice of extraction.

2D sky model

Future enhancement as suggested from the MAAT science kick-off meeting.

Error when inspecting non-flux-calibration spectrum

@P-R-McWhirter reported that inspect_reduce_spectrum of a wavelength-calibrated but not flux-calibrated spectrum leads to error:

attributeError: '_spectrum1D' object has no attribute 'count_standard'

CR not rejected in optimal extraction

@rjs3273 reported that cosmic rays are not removed in optimal extraction.

better logging/warning system

Use logging from the Python Standard Library, which is already used in RASCAL.

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Support user supplied line list

Most observatories provide the arc line list for their lamps, such information can massively improve the reliability of automated wavelength calibration.

sensitivity curve should be a polynomial fit

compute_sensitivity is using an interpolation function instead of a polynomial fit. This leads to a noisy sensitivity curve, which is significantly worse near the detector edges.

Need a plotting library that is independent of plotly-orca

plotly-orca is very unfriendly in terms of installation effort, an independent static plotting library should be used, for example, default to matplotlib and having plotly and plotly-orca become optional dependencies. Recently also come across a server not being able doing background plotting because of the lack of a display, which was solved by using Xvfb, again, it works but it is a very user unfriendly setup.

ap_trace is consistently tracing spectra below the trace

@rjs3273 reported that the trace found from ap_trace is consistently shifted below the spectrum (as traced by eye; below = smaller pixel number). It is not clear where the problem is at the moment. Could be a resampling issue, could be a plotting issue.

Calls to ap_trace_quick fail if plotly orca is not installed

As title - fix is:

conda install -c plotly plotly-orca

But probably an import somewhere it shouldn't be :)

nan is not handled properly in TwoDSpec

Reported by @P-R-McWhirter, when a numpy.array is provided instead of an ImageReduction object, the TwoDSpec is not handling nan properly and lead to error.

Improve test coverage

At the time of starting this issue, the coverage is only 63%. The goal is not to achieve 100% in a short timescale but to at least cover all the frequently used functions.

https://coveralls.io/github/cylammarco/ASPIRED

Support user supplied sensitivity curve (wavelength)

This is necessary if the service does not provide standards but only the sensitivity curve( as a function of wavelength).

Transcribing extensive FITS headers from the raw data

Just starting a discussion.

Currently ASPIRED generates FITS with a bare minimal header. The input files of course include extensive data about the observation circumstances, telescope and the instrument. In order to integrate the data into a complete workflow there needs to be some way of propagating all those data through into the headers of the resulting data. As a trivial example, I have just reduced a couple of hundred SPRAT frames through ASPIRED, but now have no easy way of telling which were obtained in the BLUE config and which in the RED.

So my questions is whether there is any plan for this sort of feature in ASPIRED and also whether it makes any sense anyway?

It would be very straight forward to have ASPIRED transcribe everything from the input FITS file, but you could easily end up with invalid FITS headers. If ASPIRED just copies the full header for some arbitrary instrument it has no way of knowing whether or not those values make sense after extraction.

You could have an option to transcribe certain nominated values to the output FITS, but it might end up an enormous list that is difficult to keep up to date.

Is it better to just say that the user needs to write all that into their own wrapper and handle it for themselves. ASPIRED only ever creates a barest minimum header of parameters that it controls itself.

If the latter, then the pipeline workflow is something like

• Run ASPIRED
• Save ASPIRED result into a temporary file on disk
• Create your own output file with your own bespoke headers
• Transcribe data from the temporary aspired file into your final file.
  This seems a bit clumsy, but it is flexible and very general, allowing you do format your own outputs however you like.

ImageReduction specification input list requires arc

It appears to be mandatory that the text list file passed to ImageReduction() includes an arc, but it is not tested and trapped if missing. I wanted to use the tracing and extraction routines without doing a wavelength calibration so I did not include an arc and got the following error.

---

IndexError Traceback (most recent call last)
in
----> 1 science_frame = aspired.ImageReduction('RJSTest/v_e_20160808_20_1_0_1.list');

/data/LT/Commissioning/SPRAT/Aspired/ASPIRED/aspired/aspired.py in init(self, filelistpath, ftype, saxis, saxis_keyword, combinetype_light, sigma_clipping_light, clip_low_light, clip_high_light, exptime_light, exptime_light_keyword, combinetype_dark, sigma_clipping_dark, clip_low_dark, clip_high_dark, exptime_dark, exptime_dark_keyword, combinetype_bias, sigma_clipping_bias, clip_low_bias, clip_high_bias, combinetype_flat, sigma_clipping_flat, clip_low_flat, clip_high_flat, silence)
239 dtype='str',
240 autostrip=True)
--> 241 self.imtype = self.filelist[:, 0]
242 self.impath = self.filelist[:, 1]
243 try:

IndexError: too many indices for array

After adding an arc line to the text file with a dummy file name assigned to "arc" that I will never use it works OK.

Descriptive labels in FITS extensions

The final output FITS from OneDSpec.savefits() can have up to about 15 extensions. I am struggling to tell them all apart and what is in each. Could description labels be added to each FITS header?

If I use
OneDsSpec.save_fits(output='flux_resampled')
then I get four HDUs. Three are labelled "flux" and one is labeled "sensitivity". When I add more options into the output='' string then I get no labels at all. The headers at the moment are extremely sparse and I assume that is a work in progress.

The S/N plot looks wrong

The S/N shown in the ap_extract plot seems too small, perhaps an extra/missed a square root somewhere?

Support user supplied wavelength calibration polynomial coefficients

Quicklook rapid reduction for data from a moderately stable instrument can use a fixed set of polynomial coefficients to speed up the reduction time for immediate data inspection.

ap_trace fails to fit the profile of saturated traces

ap_trace() can trace a saturated spectrum, however, the LSF is not usable.

Reduced frame should not overwrite self.light_master

As reported by @P-R-McWhirter, the fieldflattening should not apply directly to the self.light_master, a new frame, e.g. self.light_reduced, should be created from the raw light/flat/dark/bias frames.

ASPIRED/aspired/image_reduction.py

Line 518 in ec5f303

def reduce(self):

Typo in the installation instructions

It's a pretty trivial typo, but being in the "how to install" instructions, it is going to cause confusion. If I didn't happen to know your real name it would probably have had me stumped for rather longer.

On
https://aspired.readthedocs.io/en/latest/installation/pip.html
you have
https://github.com/cyalmmarco/ASPIRED.git
instead of
https://github.com/cylammarco/ASPIRED.git

Difficult to load standard star by name

Need to improve the search and guess facility of StandardFlux.

Rectification axis 1 is out of bounds

There should be a handler to handle with out of bound sub-spectra indices when performing cross-correlation.

Install without the test data

I appreciate this is probably an administrative headache to reorganise the repository, but is there some easy mechanism by which I can install updates without having to pull the many Mb of test data? I am using
pip install git+https://github.com/cylammarco/ASPIRED@dev
and it takes about 15min. I know it would also be possible to clone the repository once and pull updates, but you previously suggested that the pip install was the safer option until the other setup.py issues get cleaned up. Basically, does this require reorganising the repository or is there some pip magic I can use from my end to save myself downloading all those FITS? 15min is not the end of the world, so this is not a huge issue. Feel free to say I just have to live with it for the time being!

WCS in adu_resampled FITS extensions

When I save with
OneDSpec.save_fits(output='flux_resampled+wavecal+flux+adu+adu_resampled')
I was interpreting flux_resampled and adu_resampled as being the same size arrays and the same wavelength coordinates. Is that correct? Certainly they are the same dimensions. By contrast 'adu' has the same dimensions as the original CCD without resampling.

SO my question is, is the WCS in 'adu_resampled' the same as 'flux_resampled'. Can I simply read the WCS from 'flux_resampled' and use if for 'adu_resampled'? Will the WCS eventually be written into 'adu_resampled' too?

Flat frames are not bias and dark subtracted

Reported by @P-R-McWhirter, the flat frames are not bias and dark subtracted in the ImageReduction.

Support flux calibration for user supplied wavelength calibrated science and standard spectra

Data could be reduced by the facilities in a per-observing block basis, in that case, ASPIRED only needs to handle wavelength calibrated data.

Version specific dev branch

Still at the early development stage, however, we should start separating the current version dev/maintenance branch (for patch release) and a dev branch for minor releases.

pip struggles to install ASPIRED

Reported from Ilknur:

pip install git+https://github.com/cyalmmarco/ASPIRED.git@dev installs the dist-info but not the package

however
git clone https://github.com/cyalmmarco/ASPIRED.git@dev then pip install -e ASPIRED works

Is it a problem with pip or with setup.py? Probably the latter?

ap_extract fails near edge

ap_extract may not work near the edges of an image, it definitely does not work near edges in force-extraction mode.

Attempt to handle string to numeric type conversion when reading FITS header

exposure time, gain, exposure time, readnoise, humidity, air pressure, temperature etc.

Optimal extraction for curved spectra

See Marsh 1989.

Multithreading/Multiprocessing/GPU for speed up?

This was brought up for a few times in various meetings and seminars. Should consider this in the infrastructure design when expanding to (stably) handle multiple spectral extractions.

OneDSpec save fits/csv does not accept 'wavelength' as an output option.

When calling the save_fits or save_csv routines of a OneDSpec object, requesting a wavelength output is not fulfilled. Additionally, this blocks the return of any other output option (flux, count, trace, etc.).

FITS header not handled in `ImageReduction`

Reported by @P-R-McWhirter, header is not handled in ImageReduction

Cannot save spectrum in instrumental coordinates (counts vs pix)

This is a pretty trivial one. It's not something I really absolutely need at the moment so it's not urgent.

I tried to use
oneDSpec.save_fits(output='count')
to save off a copy of the basic extraction in instrumental units, I.e., just photo-electrons (or counts) against CCD pixel. I got the error message.

Traceback (most recent call last):
File "/data/lt/Commissioning/SPRAT/Aspired/test_aspired_July2020.py", line 130, in
overwrite=True)
File "/Users/rjs/miniconda3/envs/astroconda/lib/python3.6/site-packages/aspired/spectral_reduction.py", line 7630, in save_fits
raise Error('Neither wavelength nor flux is calibrated.')
NameError: name 'Error' is not defined

After I have done the wavelength and flux calibrations, then save_fits() works and will indeed save off a file in exactly the format I wanted, but it seems a bit odd to insist that you must wavelength calibrate the data before you can save off the uncalibrated version. As I say, I can certainly live without it for now, but thought I'd point it out.

set_known_pairs() does not work

Some API issue or rascal.set_known_pairs() does not work.

Frame masking for reduction and extraction

1. ImageReduction should be able to generate a good/bad pixel mask or to accept an external mask.

2. TwoDSpec should be able to accept a mask.

ING standard stars cannot be loaded

file name and extension problems.

3 copies of the header are added to the output fits file

It is reported that 3 copies of the same header are added to the output fits file.

fit_coeff param error in WavelengthCalibration.fit()

Calling WavelengthCalibration.fit() with the optional display plot switch on is giving me an error for parameter fit_coeff.

I think the issue is that fit_coeff does not exist as a parameter in rascal.calibrator.plot_fit(). Possibly I just have the wrong rascal? I (think I) am running the rascal dev branch.

target1D.fit(max_tries=50, stype='science+standard', display=True)

Traceback (most recent call last):
File "/data/lt/Commissioning/SPRAT/Aspired/test_aspired_July2020.py", line 153, in
target1D.fit(max_tries=50, stype='science+standard', display=args.displayplots)
File "/Users/rjs/miniconda3/envs/astroconda/lib/python3.6/site-packages/aspired/spectral_reduction.py", line 7115, in fit
filename=filename)
File "/Users/rjs/miniconda3/envs/astroconda/lib/python3.6/site-packages/aspired/spectral_reduction.py", line 4091, in fit
filename=filename)
TypeError: plot_fit() got an unexpected keyword argument 'fit_coeff'

compute_sencurve(), spectres() limit bound error

Just in order to get an idea of how a basic reduction would flow, I have been running the included Jupyter notebooks. They seem to work fine for me up to compute_sencurve() where they generate an exception in spectres.

Python v3.7.4
ASPIRED version: git cloned today 2019-09-13
spectres v2.0.0

---

ValueError Traceback (most recent call last)
in
2 lhs6328_reduced = aspired.OneDSpec(lhs6328, wavecal, standard=hilt102, wave_cal_std=wavecal, flux_cal=fluxcal)
3 lhs6328_reduced.apply_wavelength_calibration('all')
----> 4 lhs6328_reduced.compute_sencurve(kind='cubic')
5 lhs6328_reduced.inspect_sencurve()

/data/LT/Commissioning/SPRAT/Aspired/ASPIRED/aspired/aspired.py in compute_sencurve(self, kind, smooth, slength, sorder, display)
1414 # resampling both the observed and the database standard spectra
1415 # in unit of flux per second
-> 1416 flux_std = spectres(self.wave_std_true, self.wave_std, self.adu_std / self.exptime_std)
1417 flux_std_true = self.fluxmag_std_true
1418 else:

~/miniconda2/envs/astroconda/lib/python3.7/site-packages/spectres/spectral_resampling.py in spectres(new_spec_wavs, old_spec_wavs, spec_fluxes, spec_errs)
70 # Check that the range of wavelengths to be resampled_fluxes onto falls within the initial sampling region
71 if filter_lhs[0] < spec_lhs[0] or filter_lhs[-1] > spec_lhs[-1]:
---> 72 raise ValueError("spectres: The new wavelengths specified must fall within the range of the old wavelength values.")
73
74 #Generate output arrays to be populated

ValueError: spectres: The new wavelengths specified must fall within the range of the old wavelength values.

Change master to main

Rename branch name to the new GitHub default naming.

Can save_fits() write out spectra that are not yet flux calibrated?

save_fits() calls _create_fits() which assumes the flux calibration has already been applied. I believe users will frequently want to save off the spectrum at various stages along the reduction. In particular it will be extremely common for observers to extract and wavelength calibrate an observation but not have a standard available. Even if they do intend to flux calibrate, they may simply have observed the target before the standard.

Support user supplied wavelength calibrated standard

Multiple observations may depend on the same set of standard observation for calibration, it is massively inefficient if the standard has to be recalibrated in each reduction.

More efficient handling of _spectrum1D

The current design produces duplicated copies of _spectrum1D objects in onddspec.wavecal and onedspec.fluxcal. We should design a structure to allow shared _spectrum1D objects so that both wavecal and fluxcal are accessing and modifying the same memory to avoid duplication error.

Support user supplied pixel-to-wavelength mapping (not fitted)

Similar to #16, but also provide the pixel values of the lines, which will be used as anchors. This will improve the calibration quality even more than #16. However, if wrong pixel-to-wavelength mapping is provided, it will make the fit worse.

Discussion of defaults on inspect_reduced_spectrum()

What is the proper venue for discussions? The fact this page is called "issues" sort of implies something is wrong. I would not go that far. I just want to raise a question about selecting parameter defaults.

Currently we have

inspect_reduced_spectrum(self, wave_min=4000., wave_max=8000., ...

That is awfully SPRAT specific. Whilst that is obviously best for me and makes my life easier playing with SPRAT data, it seems to me that better defaults would either a) the entire detector array or b) some clever automated guess based on, for example where the ap_trace fails and drops to zero. I would suggest that a is enough for release v1 and you can add b to the ideas list.

Filepath is not trapped byTwoDSpec()

Filepath to an arc frame is not trapped:

ASPIRED/aspired/spectral_reduction.py

Line 3688 in ec5f303

raise TypeError(

which is implemented for the light-frame:

ASPIRED/aspired/spectral_reduction.py

Line 1778 in ec5f303

elif isinstance(data, str):