mpi-astronomy/snowblind

Algorithms for cleaning JWST data

Algorithms for cleaning JWST data.

• SnowblindStep: mask cosmic ray showers and snowballs
• JumpPlusStep: Propagate JUMP_DET and SATURATED flags in GROUPDQ properly for frame-averaged groups
• PersistenceFlagStep: flag pixels effected by persistence exposure-to-exposure
• OpenPixelStep: flag new open pixels, hot pixels, or open adjacent pixels via self-cal

Installation

pip install snowblind

Usage

The steps in snowblind run like any other pipeline steps. From the command line you can run SnowblindStep (aliased as snowblind) on the result file from JumpStep:

strun snowblind jw001234_010203_00001_nrcalong_jump.fits --suffix=snowblind

Or you can run SnowblindStep and JumpPlusStep as post-hooks after JumpStep in a full pipeline, remembering to turn off the default snowball flagging.

strun calwebb_detector1 jw001234_010203_00001_nrcalong_uncal.fits --steps.jump.post_hooks="snowblind.SnowblindStep","snowblind.JumpPlusStep" --steps.jump.flag_large_events=False

In Python, we can insert SnowblindStep and JumpPlusStep after JumpStep as a post-hook:

from snowblind import SnowblindStep, JumpPlusStep
from jwst.pipeline import Detector1Pipeline


steps = {
    "jump": {
        "save_results": True,
        "flag_large_events": False,
        "post_hooks": [
            SnowblindStep,
            JumpPlusStep,
        ],
    },
}

Detector1Pipeline.call("jw001234_010203_00001_nrcalong_uncal.fits", steps=steps, save_results=True)

More to come on the other steps available.

snowblind's Issues

Rename RcSelfCalStep

Rename RcSelfCalStep to something else. These are not RC pixels. Mostly they are what is referred to as OPEN pixels, i.e. any flux landing on them gets recorded in adjacent pixels. Bit value for these OPEN pixels is currently 26. And adjacent pixels that receive the charge are ADJ_OPEN, bit value 27.

Find saturated pixels only inside jump detections

Use the methods described here

https://stackoverflow.com/questions/70517818/how-do-skimage-morphology-remove-small-holes-and-skimage-morphology-remove-small

To find SATURATED pixels only within JUMP_DET pixels, and these are the ones that get expanded. Currently all SATURATED + JUMP_DET pixels get expanded.

See if this improves the false positives when the snowballs are smaller for NIRSpec.

Set 'COMPLETE' in the FITS header

Currently when a step is 'COMPLETE', an attribute in the datamodel is updated, but since the schema does not map this to keyword, this is not stored in the FITS header, and when the file is reopened, it is lost. Update the FITS header instead, and figure out a fix upstream in stdatamodels and stpipe that would allow this to work properly.

Some real sources are masked while the jump step with `expand_large_events=True` in the JWST pipeline does not.

I found that in this specific image sets jw01837004001_04201_00001 and jw01837004001_04201_00002 (F356W, four images) snowblind does not work well. The centers of many sources are masked. But this does not happen in other images I have reviewed. Only in this image. And this problem does not happen in the JWST Stage-1 pipeline with the argument of the jump step expand_large_events set to True.

Download link for the file
https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/jw01837004001_04201_00001_nrcalong_uncal.fits
https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/jw01837004001_04201_00001_nrcblong_uncal.fits
https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/jw01837004001_04201_00002_nrcalong_uncal.fits
https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/jw01837004001_04201_00002_nrcblong_uncal.fits

The code to reproduce this problem and the images below

# jwst version: 1.12.5
# CRDS context: jwst_1166.pmap
from jwst.pipeline import Detector1Pipeline, Image2Pipeline

file = 'jw01837004001_04201_00002_nrcalong_uncal.fits'

# Pipeline with `expand_large_events` set to True
args = {'persistence': {'save_trapsfilled': False},
        'jump': {'skip': False,
        'expand_large_events': True}}
rate_pipeline = Detector1Pipeline.call(file, steps=args)
cal_pipeline = Image2Pipeline.call(rate_pipeline, steps={'resample': {'skip': True}})

# Snowblind
args = {'persistence': {'save_trapsfilled': False},
        'jump': {'skip': False,
        'expand_large_events': False,
        'post_hooks': ['snowblind.SnowblindStep']}}
rate_snowblind = Detector1Pipeline.call(file, steps=args)
cal_snowblind = Image2Pipeline.call(rate_snowblind, steps={'resample': {'skip': True}})

Write version of snowblind to data header

Version of snowblind is not recorded in the FITS output header. Do it.

Laplace filter self-cal stacks to find OPEN pixels

After some discussion with the Euclid NISP folks, it may make sense to use a 2D Laplace kernel to make OPEN and ADJ_OPEN pixels stand out more in OpenPixelStep. Convolving with a kernel something like:

[[0,  1, 0],
 [1, -4, 1],
 [0,  1, 0]]

before thresholding will make them stand out more.

Stricter flagging of bad pixel neighbors

Another tweak that I have made while looking at NIRCam bad pixels is to look at neighbors of pixels flagged by RCSelfCal using a stricter threshold.

I've seen the best results so far by running RCSelfCal with a 5 sigma threshold, and then going back and looking at the 4 neighbors around each flagged pixel and flagging those using a 3 sigma threshold. I currently do this neighbor flagging outside of RCSelfCal, but I could look into integrating it if you think it would be useful.

Add snowball rate to header of corrected files

There's a header keyword EXTNCRS that records this in the schema. Populate it.