ijiraq / gemini_processing Goto Github PK
View Code? Open in Web Editor NEWscripts and stuff to process gemini data witin the CADC.
scripts and stuff to process gemini data witin the CADC.
Creating this issue to record the decision from Tuesday's meeting.
Hi JJ & all: To be consistent with the way we save the source for other CADC managed containers on the science platform, I've copied gemini_processing
to opencadc/science-platform/containers/software-containers/gemini
. I think this repo should be deleted at some point. If there are any relevant open issues they should be ported over.
Utilize the CADC TAP service and CAOM content to gather the NIFS files needed for processing.
After finalizing product names in #31 , make sure products on Cavern can be found properly for ingestion (with help from David and Sharon, and possibly Ling).
Issue Nat1405/niriPipe#10 needs to build a Dockerfile from a single source of truth. It makes sense to store that Dockerfile and Makefile here in gemini_processing.
This issue is to create a Dockerfile that:
make
nat1405/niriPipe
dev
branch of nat1405/niriPipe
on each build (dev
branch for now, possibly main
branch later on)pytest .
fairly easily from GitHub actions and my own personal machine.There are a few extra bits that need to be done to all a container to execute in Arcade.
Some questions about testing that will be developed later.
Determine how to use Docker to build a container and install the software environment needed to process NIFTS data.
To check the feasibility of archiving processed NIRI imaging data, produce a test set "by hand" and add appropriate metadata for CAOM2 ingestion.
We'd like to have the NIFTY that we are using not diverge from the one that GEMINI might point to. To achieve that we'd like to issue a PR to the NIFTY origin. First confirm with Marie that she is interested in this.
For the record, in case anyone is wondering about running Gemini DRAGONS in parallel:
NIRI imaging processing with DRAGONS seems to work fairly well with the standard python multiprocessing
library. However, a few steps require SExtractor and seem to be writing to temp folders that disappear in between steps. The relevant error message is something like:
2021-02-25 13:01:18 STATUS - PRIMITIVE: detectSources
2021-02-25 13:01:18 STATUS - ------------------------
2021-02-25 13:01:19 ERROR - ERROR - Reduce received an unhandled exception. Aborting ...
2021-02-25 13:01:19 ERROR - ERROR - Traceback (most recent call last):
2021-02-25 13:01:19 ERROR - ERROR - File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/recipe_system/reduction/coreReduce.py", line 229, in runr
2021-02-25 13:01:19 ERROR - ERROR - recipe(p)
2021-02-25 13:01:19 ERROR - ERROR - File "/Users/comeaun/hia_2020/reductions/niri/tim_davidge/GN-2007B-Q-85/exp_firsttry/myrecipelibrary.py", line 86, in reduce_w_pattern_noise_standard
2021-02-25 13:01:19 ERROR - ERROR - p.detectSources(stream='skysub')
2021-02-25 13:01:19 ERROR - ERROR - File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/recipe_system/utils/decorators.py", line 155, in gn
2021-02-25 13:01:19 ERROR - ERROR - ret_value = fn(pobj, adinputs=adinputs, **dict(config.items()))
2021-02-25 13:01:19 ERROR - ERROR - File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/geminidr/core/primitives_photometry.py", line 214, in detectSources
2021-02-25 13:01:19 ERROR - ERROR - sex_task.run()
2021-02-25 13:01:19 ERROR - ERROR - File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/gempy/gemini/eti/sextractoreti.py", line 106, in run
2021-02-25 13:01:19 ERROR - ERROR - self.clean()
2021-02-25 13:01:19 ERROR - ERROR - File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/gempy/eti_core/eti.py", line 142, in clean
2021-02-25 13:01:19 ERROR - ERROR - fil.clean()
2021-02-25 13:01:19 ERROR - ERROR - File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/gempy/gemini/eti/sextractoretifile.py", line 75, in clean
2021-02-25 13:01:19 ERROR - ERROR - os.remove(self._disk_file)
2021-02-25 13:01:19 ERROR - ERROR - FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/__/kmtshyr526n5xz4k_x3qgjs80000gq/T/N20080121S0176_skySubtracted_1.fits'
It's probably possible to wrap this in a try/except or get parallelism working with a minimum amount of effort; it's just not quite working perfectly yet out of the box.
In preparation for #37 , finish your NIRI data reduction experiments off.
The project is using a container environment to install the nifty package so that container can be used in Arcade for interactive mode. But we also want to do batch processing, to achieve that we could run Docker in a VM that is the launched via the regular batch scheduler.
This course might help you learn how to do this: https://wgschool.netlify.app/docker is being offered the week of June 29th. Register here: https://wgschool.netlify.app/attendee/
After talking with Stephen, a good idea for checking the quality of NIRI stacks is to compare the expected limiting magnitude from the NIRI ITC to the observed limiting magnitude. Give it a try!
We need help setting up QA process. Marie at Gemini should be able to help with that. Get in-touch and find out.
Tim is available from Feb 24th, 2021 onwards to provide NIRI processing expertise. Get your experiments cleaned up and ready before then, so you can present your work and what you've tried to him.
Before my end of term, so he and/or Nat 2.0 knows how to get the data :)
Do some rough estimates of the resources required per science program (GB per reduction) based on a few input parameters such as num_targets, num_science_frames, etc.
Gemini downloads already do an MD5 verification step; implement this for the CADC downloads.
Use and supplement Gemini DRAGONS to produce a data reduction Wes Fraser is happy with.
The new vos
package allows setting the VOSpace destination using a config.
modify the scripts to do the processing on a local disk but then copy the resulting file to a VOSpace area, the VOSpace should be set configured inside the VM or container.
Get it written up!
To distribute the outputs of the processing we will need to have the metadata stored in the CADC's caom data model. Likely this can be largely a modification of the 'raw' caom map.
End goal: process > 80% of the NIRI archive. A high development-deployment cadence might help with this. Prototype the infrastructure you'll need for this.
Any any other things you think you might need. Reuse as many (useful) pieces as you can from Nifty4Gemini.
We should look into tagging NIFS datasets with the errors/messages generated during processing, such as WavelengthError
, CalibrationsNotFound
, etc. This could be flagging existing planes as not processable, or creating processed planes marked as JUNK
to keep track of processing (this is reply to a suggestion from JJ).
IRAF-based Nifty based processing seems to no longer work on MacOS Catalina. Changes to Nifty still need tested, so we need containers to build quicker now. Fix the Dockerfile to do this, and also work in Seb's feedback on the container at the same time.
Build a container that can run Gemini DRAGONS. This should work for NIRI imaging data reduction to start.
We need to have a set of proposalIDs that span some of the scope for the kinds of NIFS data that we might be processing. Examples might be:
Or, we could just randomly sample 10% of the programs to get a representative test set.
Build a container that allows DRAGONS reductions to happen in ARCADE via Jupyter notebooks.
So far graphical IRAF tasks aren't working in the Arcade version of Nifty. They are working in a non-container version of Nifty though. @ijiraq do we want graphical tasks in Arcade?
I probably already have the fix committed, but need to re-run processing and make sure provenance information is present on all output products on VOSpace.
We need to run a few 100 jobs with this processing. To do this we can build a VM that has docker on it and then run these jobs as docker run
jobs on that VM.
Create some integrations tests based on the representative sample of NIFS proposal ID values (see issue #10 for details)
To test cross-reporitory issue automation.
The CADC archive system uses the CAOM2 data mode to record metadata about observations. Those can be queried using cadc-tap. Adapt the NIFTY data flow to use that query system and cadc-data to retrieve that files from the CADC archive.
Currently, nifty reduces an entire program by its ID. It would be nice that a failure on one target didn't affect all other targets in the reduction.
We fixed this earlier, but I'm recording the solution here for completeness. There was a problem with both the Nifty container running in ARCADE and the build process for it. The problem has been fixed, and I have updated the makefile to reflect the build steps that I actually use.
Data processed at CADC shouldn't live in VOSpace permanently. Find a new archive/database for it.
With Wes's feedback, make histograms of the time deltas between exposures. If it's bimodal, maybe we have a way to throw out the bad "first frames".
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.