ijiraq/gemini_processing

scripts and stuff to process gemini data witin the CADC.

gemini_processing's Issues

Finalize naming of CADC-generated products

Creating this issue to record the decision from Tuesday's meeting.

Moved the gemini containers to opencadc/science-platform

Hi JJ & all: To be consistent with the way we save the source for other CADC managed containers on the science platform, I've copied gemini_processing to opencadc/science-platform/containers/software-containers/gemini. I think this repo should be deleted at some point. If there are any relevant open issues they should be ported over.

Get data information from CADC services

Utilize the CADC TAP service and CAOM content to gather the NIFS files needed for processing.

Prep Nifty products for gemProc2Caom2

After finalizing product names in #31 , make sure products on Cavern can be found properly for ingestion (with help from David and Sharon, and possibly Ling).

Create a NIRI data processing container.

Issue Nat1405/niriPipe#10 needs to build a Dockerfile from a single source of truth. It makes sense to store that Dockerfile and Makefile here in gemini_processing.

This issue is to create a Dockerfile that:

• Will be built from a reliable Makefile with the command make
• Installs required dependencies for nat1405/niriPipe
• Quickly installs the latest version of code from the dev branch of nat1405/niriPipe on each build (dev branch for now, possibly main branch later on)
• Works on both Arcade and Batch
• Can run pytest . fairly easily from GitHub actions and my own personal machine.

Modify the NIFTY container to work inside CADC Arcade

There are a few extra bits that need to be done to all a container to execute in Arcade.

Work Term Tasks

Some questions about testing that will be developed later.

• Learn to build a container that can run the current version of nifty
• Build said conatiner.
• Modify nifty to use the cadc-tap system to query files.
• Use cadc-data to get those files. < - then orgnize using nifty script
• Get the container loaded into ARCADE with a cavern storage allocation.
• Use the new script to inside ARCADE to process some data.
• Work on Data Engineering to allow data to go into CAOM2.
• Give end of work term presentation.

Build a NIFTY container

Determine how to use Docker to build a container and install the software environment needed to process NIFTS data.

Produce NIRI Imaging Test Set for gemProc2caom2.

To check the feasibility of archiving processed NIRI imaging data, produce a test set "by hand" and add appropriate metadata for CAOM2 ingestion.

Isue a Pull Request to have CADC option for NIFTY pipeline.

We'd like to have the NIFTY that we are using not diverge from the one that GEMINI might point to. To achieve that we'd like to issue a PR to the NIFTY origin. First confirm with Marie that she is interested in this.

Parallelism in DRAGONS

For the record, in case anyone is wondering about running Gemini DRAGONS in parallel:

NIRI imaging processing with DRAGONS seems to work fairly well with the standard python multiprocessing library. However, a few steps require SExtractor and seem to be writing to temp folders that disappear in between steps. The relevant error message is something like:

2021-02-25 13:01:18 STATUS   -    PRIMITIVE: detectSources
2021-02-25 13:01:18 STATUS   -    ------------------------
2021-02-25 13:01:19 ERROR    - ERROR - Reduce received an unhandled exception. Aborting ...
2021-02-25 13:01:19 ERROR    - ERROR - Traceback (most recent call last):
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/recipe_system/reduction/coreReduce.py", line 229, in runr
2021-02-25 13:01:19 ERROR    - ERROR -     recipe(p)
2021-02-25 13:01:19 ERROR    - ERROR -   File "/Users/comeaun/hia_2020/reductions/niri/tim_davidge/GN-2007B-Q-85/exp_firsttry/myrecipelibrary.py", line 86, in reduce_w_pattern_noise_standard
2021-02-25 13:01:19 ERROR    - ERROR -     p.detectSources(stream='skysub')
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/recipe_system/utils/decorators.py", line 155, in gn
2021-02-25 13:01:19 ERROR    - ERROR -     ret_value = fn(pobj, adinputs=adinputs, **dict(config.items()))
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/geminidr/core/primitives_photometry.py", line 214, in detectSources
2021-02-25 13:01:19 ERROR    - ERROR -     sex_task.run()
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/gempy/gemini/eti/sextractoreti.py", line 106, in run
2021-02-25 13:01:19 ERROR    - ERROR -     self.clean()
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/gempy/eti_core/eti.py", line 142, in clean
2021-02-25 13:01:19 ERROR    - ERROR -     fil.clean()
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/gempy/gemini/eti/sextractoretifile.py", line 75, in clean
2021-02-25 13:01:19 ERROR    - ERROR -     os.remove(self._disk_file)
2021-02-25 13:01:19 ERROR    - ERROR - FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/__/kmtshyr526n5xz4k_x3qgjs80000gq/T/N20080121S0176_skySubtracted_1.fits'

It's probably possible to wrap this in a try/except or get parallelism working with a minimum amount of effort; it's just not quite working perfectly yet out of the box.

Finish NIRI Reduction Experiments

In preparation for #37 , finish your NIRI data reduction experiments off.

Run docker on a VM to allow batch processing based on container

The project is using a container environment to install the nifty package so that container can be used in Arcade for interactive mode. But we also want to do batch processing, to achieve that we could run Docker in a VM that is the launched via the regular batch scheduler.

This course might help you learn how to do this: https://wgschool.netlify.app/docker is being offered the week of June 29th. Register here: https://wgschool.netlify.app/attendee/

Check limiting magnitude of NIRI stacks.

After talking with Stephen, a good idea for checking the quality of NIRI stacks is to compare the expected limiting magnitude from the NIRI ITC to the observed limiting magnitude. Give it a try!

Find some QA assistances

We need help setting up QA process. Marie at Gemini should be able to help with that. Get in-touch and find out.

Get data processing feedback from Tim Davidge

Tim is available from Feb 24th, 2021 onwards to provide NIRI processing expertise. Get your experiments cleaned up and ready before then, so you can present your work and what you've tried to him.

Document for David how to access NIFS/NIRI data from Arcade

Before my end of term, so he and/or Nat 2.0 knows how to get the data :)

Estimate NIFS program reduction resources

Do some rough estimates of the resources required per science program (GB per reduction) based on a few input parameters such as num_targets, num_science_frames, etc.

MD5 Checksum of Downloads

Gemini downloads already do an MD5 verification step; implement this for the CADC downloads.

Get Wes satisfied with the reduction of NIRI GN-2019A-FT-108

Use and supplement Gemini DRAGONS to produce a data reduction Wes Fraser is happy with.

Need a process for persisting data to VOSpace that is configurable.

The new vos package allows setting the VOSpace destination using a config.

modify the scripts to do the processing on a local disk but then copy the resulting file to a VOSpace area, the VOSpace should be set configured inside the VM or container.

Finish ADASS submission for NIFS processing

Get it written up!

Create CAOM data engineering map for NIFS processed data.

To distribute the outputs of the processing we will need to have the metadata stored in the CADC's caom data model. Likely this can be largely a modification of the 'raw' caom map.

Build a skeleton NIRI system

End goal: process > 80% of the NIRI archive. A high development-deployment cadence might help with this. Prototype the infrastructure you'll need for this.

• Configuration management via git and Docker, requirements.txt, etc.
• Tests via Travis CI
• Data persistence process
• Some sort of tests or scripts to get final results of processing (pipeline diagnostics, VOSpace level checks, etc).

Any any other things you think you might need. Reuse as many (useful) pieces as you can from Nifty4Gemini.

'JUNK' planes for unprocessable datasets

We should look into tagging NIFS datasets with the errors/messages generated during processing, such as WavelengthError, CalibrationsNotFound, etc. This could be flagging existing planes as not processable, or creating processed planes marked as JUNK to keep track of processing (this is reply to a suggestion from JJ).

Refactor Nifty Docker Container

IRAF-based Nifty based processing seems to no longer work on MacOS Catalina. Changes to Nifty still need tested, so we need containers to build quicker now. Fix the Dockerfile to do this, and also work in Seb's feedback on the container at the same time.

Make sure NIRI products are showing up in GEMINIPROC rather than GEMINI

Run gemProc2Caom2 ingestion

I believe this is a task for Ling once we get everything hooked up? We need to resolve #31 and #32 first.

Gemini Dragons Container

Build a container that can run Gemini DRAGONS. This should work for NIRI imaging data reduction to start.

Create a list of representative NIFS proposalIDs

We need to have a set of proposalIDs that span some of the scope for the kinds of NIFS data that we might be processing. Examples might be:

• Long exposures
• bright targets
• large numbers of targets.

Or, we could just randomly sample 10% of the programs to get a representative test set.

Gemini DRAGONS notebook container

Build a container that allows DRAGONS reductions to happen in ARCADE via Jupyter notebooks.

Graphics in Nifty Container

So far graphical IRAF tasks aren't working in the Arcade version of Nifty. They are working in a non-container version of Nifty though. @ijiraq do we want graphical tasks in Arcade?

Fix nifty4gemini provenance issues

I probably already have the fix committed, but need to re-run processing and make sure provenance information is present on all output products on VOSpace.

Run the container in batch mode

We need to run a few 100 jobs with this processing. To do this we can build a VM that has docker on it and then run these jobs as docker run jobs on that VM.

Create tests of the NIFTY container based on representative sets.

Create some integrations tests based on the representative sample of NIFS proposal ID values (see issue #10 for details)

Test Issue

To test cross-reporitory issue automation.

Modify nifty to use the cadc-tap system to query files.

The CADC archive system uses the CAOM2 data mode to record metadata about observations. Those can be queried using cadc-tap. Adapt the NIFTY data flow to use that query system and cadc-data to retrieve that files from the CADC archive.

Have nifty run by target as well as by program ID

Currently, nifty reduces an entire program by its ID. It would be nice that a failure on one target didn't affect all other targets in the reduction.

Fix bug in ARCADE found by JJ on Nov 6th.

We fixed this earlier, but I'm recording the solution here for completeness. There was a problem with both the Nifty container running in ARCADE and the build process for it. The problem has been fixed, and I have updated the makefile to reflect the build steps that I actually use.

Create new database for CADC Processed Data

Data processed at CADC shouldn't live in VOSpace permanently. Find a new archive/database for it.

NIRI first frame: Try Wes's suggestion.

With Wes's feedback, make histograms of the time deltas between exposures. If it's bimodal, maybe we have a way to throw out the bad "first frames".