tomasstolker/pycrires

Data reduction pipeline for VLT/CRIRES+

pycrires's Issues

Master flat no longer correctly included in SOF

The flat correction was not applied by methods such as util_calib and obs_nodding because the UTIL_MASTER_FLAT key is no longer supported in the SOF file by the recent EsoRex recipes. Instead, only the CAL_FLAT_MASTER key can be used.

This issue has been fixed in commit 00cb0f3.

The commit also implements the create_sof parameter in all methods that use a SOF file. When setting to False, a previously created SOF is not overwritten so it can be manually adjusted.

Incorrect ordering of multiple exposures due to behavior of glob

In my version of python and with my filesystem, the glob function returns file names in an incorrect ordering (see example in the end, which is the 'fits_item' generator converted to a list). Since naming output is based on the fits_idx index generated by enumerate, the spectra result reshuffled after the analysis.

I have noticed this behavior only in fit_gaussian, while in other functions the order is correctly enforced.

Proposed fix (inelegant, but is compatible with the rest of the code):

        fits_files = list(Path(input_folder).glob(
            f"cr2res_combined{nod_ab}_*_extr2d.fits"
        ))

        fits_files.sort(key=lambda x: int(x.stem.split('_')[2]))

/Users/pino/Programs_work/HD169142_appo/analysis_raw_data/product/custom_extract_2d/cr2res_combinedB_002_extr2d.fits
/Users/pino/Programs_work/HD169142_appo/analysis_raw_data/product/custom_extract_2d/cr2res_combinedB_007_extr2d.fits
/Users/pino/Programs_work/HD169142_appo/analysis_raw_data/product/custom_extract_2d/cr2res_combinedB_004_extr2d.fits
/Users/pino/Programs_work/HD169142_appo/analysis_raw_data/product/custom_extract_2d/cr2res_combinedB_008_extr2d.fits
/Users/pino/Programs_work/HD169142_appo/analysis_raw_data/product/custom_extract_2d/cr2res_combinedB_001_extr2d.fits
/Users/pino/Programs_work/HD169142_appo/analysis_raw_data/product/custom_extract_2d/cr2res_combinedB_006_extr2d.fits
/Users/pino/Programs_work/HD169142_appo/analysis_raw_data/product/custom_extract_2d/cr2res_combinedB_003_extr2d.fits
/Users/pino/Programs_work/HD169142_appo/analysis_raw_data/product/custom_extract_2d/cr2res_combinedB_000_extr2d.fits
/Users/pino/Programs_work/HD169142_appo/analysis_raw_data/product/custom_extract_2d/cr2res_combinedB_005_extr2d.fits

util_normflat missing one of the orders with latest ESO cr2res pipeline

The latest version of the cr2res pipeline misses the top order in the util_normflat recipe with the default parameters:

I ran again the pipeline on old calibrations (November 2023) and the same problem occurs, which is why I think it comes from the pipeline. I guess the problem can be easily corrected with some changes in the default parameters of the recipe, but I have not yet had the time to look into it. I create this ticket to keep track of the problem.

obs_nodding and obs_nodding_irregular do not seem to behave well with multiple exposures per nod

I have been trying to reduce an ABBA sequence where each A and B nod consists of 5 exposures. So the sequence goes:
A,A,A,A,A,B,B,B,B,B, B,B,B,B,B,A,A,A,A,A, etc.

When using pipeline.obs_nodding, it takes exposure 5 (A) and exposure 6 (B) as a nodding pair. Exposures 1 to 4 and 7-10 are ignored entirely.

When using pipeline.obs_nodding_irregular, it starts by taking exposure 1 (A) and exposure 6 (B), which I think is the desired behaviour, and then continues with exposure 2 (A) and exposure 6 (B) again, and the remaining B exposures of the A-B pair are subsequently ignored.

I believe that a solution should ideally be at the level of obs_nodding (because this sequence is regular), in that it recognises the number of A and B frames in each nod pair correctly.

The desired behaviour is (probably) that it uses exposures 1 and 6, 2 and 7, 3 and 8, 4 and 9, 5 and 10, 11 and 16, 12 and 17, etc. as nod pairs. In the case of a time-series (which is my use case), the user can then use the individually extracted A and B exposures.

Possible code indentation problem in unit testing

I am not entirely sure, but I think there are some problems in your test functions. Let's take the example of test_util_calib_fpet():

def test_util_calib_fpet(self) -> None:
    if shutil.which("esorex") is None:
        with pytest.raises(RuntimeError) as error:
            self.pipeline.util_calib(calib_type="fpet", verbose=False)
        assert str(error.value) == self.esorex_error
    else:
        self.pipeline.util_calib(calib_type="fpet", verbose=False)

I think the indentation of assert str(error.value) == self.esorex_error is not correct. This statement should be within the scope of with ..., otherwise error is not defined. This is the case in all your test functions.

However, I am not 100% sure here, so maybe it's just me! ;-)