opencadc/caom

Common Archive Observation Model

CAOM

Common Archive Observation Model

This module contains a VO-DML description of the CAOM data model and test code to validate the model using the cadc-vodml tools.

The currently released documentation generated from this description:

http://www.opencadc.org/caom2/

The Common Archive Observation Model is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.

caom's Issues

uniqueness of Artifact.uri

The Artifact.uri field is a reference to an externally stored object (usually a file, but could be a database table or maybe a directory in VOSpace holding multiple files).

It was intended that Artifact.uri is unique -- no two artifacts have the same URI -- but the implementations did not fully enforce this and usage is not consistent with the original intent.

detail:

• the java library enforces uniqueness of Artifact.uri within the parent Plane only
• validation of an Observation could enforce uniqueness of Artifact.uri within an Observation
• only a database constraint could enforce uniqueness of Artifact.uri more widely

Add lunar keywords

A number of our observatories, include space missions like WISE, record characteristics about the moon. IRTF has the most complete information

• LUN_FLI: Fraction Lunar Illumination (FLI) is the percent of the Moon's visible disk illuminated by the sun. Range is 0.0 to 100.0.

• LUN_LIGHT = The lunar light level based on the lunar elevation (EL), and Fraction Lunar Illumination (FLI) values from JPH Horizon. Values are:

  • dark = 0% <= FLI <25.0%, or Moon Elevation < 0 degrees.
  • gray = 25% <= FLI < 75.0% with Moon Elevation > 0 degrees.
  • bright = 75.0 <= FLI, and Moon Elevation > 0 degrees.

• LUN_SEP = The lunar separation in degrees of RA,DEC – moon.

• LUN_EL = The lunar position's Elevation in degrees, +90.0 to -90.0

• LUN_AZ = The lunar position's Azimuth in degrees. 0-360. 0=North, 90=east.

This seems common enough, especially for ground based observatories, such that at least some of these keywords should be supported.

more formal refence to existing IVOA vocabularies

Plane.observable.ucd -> UCD1+ spec at https://www.ivoa.net/documents/UCD1+/
Plane.dataProductType -> http://www.ivoa.net/rdf/product-type aka ObsCore datya product type
Artifact.productType -> http://www.ivoa.net/rdf/datalink/core aka DataLink semantics

Addition of *_calib_status "optional" ObsCore attributes to the model?

The ObsCore data model contains "optional" elements of the form *_calib_status, providing more information about the level of calibration of various axes, spatial, temporal, spectral, and observable. We are likely to include these in the Rubin Observatory's ObsCore tables, basically because the overall calib_level = 1 vs. =2 distinction doesn't adequately capture the way we create the planned data products, and in particular how the observable (flux-like) axis is calibrated.

In order to maintain the connection with CAOM2, we would like to suggest that the Position, Time, Energy, and Observable objects in the CAOM2 data model each be supplemented with a string-valued calib_status attribute with multiplicity [0..1] (i.e., optional).

Looking at the language in the ObsCore standard (quoted verbatim below) it doesn't seem like the enumeration values for these attributes are sufficiently standardized to be able to force them to be explicit enumerations in CAOM.

Attribute	Short title	Principal	Utype	Suggested values	Description
s_calib_status	Type of calibration along the spatial axis	1	Char.SpatialAxis .calibrationStatus	uncalibrated, raw, calibrated	A string to encode the calibration status along the spatial axis (astrometry). Possible values could be {uncalibrated, raw, calibrated} and correspond to the Utype Char.SpatialAxis.calibrationStatus. For some observations, only the pointing position is provided (s_calib_status =”uncalibrated”). Some other may have a raw linear relationship between the pixel coordinates and the world coordinates (s_calib_status = ”raw”).
t_calib_status	Type of time coordinate calibration	0	Char.TimeAxis .calibrationStatus	uncalibrated, calibrated, raw, relative	This parameter gives the status of time axis calibration. This is especially useful for time series. Possible values are principally {uncalibrated, calibrated, raw, relative}. This may be extended for specific time domain collections.
em_calib_status	Type of spectral coord calibration	0	Char.SpectralAxis .calibrationStatus	uncalibrated, calibrated, relative, absolute	This attribute of the spectral axis indicates the status of the data in terms of spectral calibration. Possible values are defined in the Characterisation Data Model and belong to {uncalibrated, calibrated, relative, absolute}.
o_calib_status	Type of calibration for the observable coordinate	1	Char.ObservableAxis .calibrationStatus	absolute, relative, normalized, any	This describes the calibration applied on the Flux observed (or other observable quantity). It is a string to be selected in {absolute, relative, normalized, any} as defined in the SSA specification (Tody, Dolensky and al. 2012) in section 4.1.2.10. This list can be extended or updated for instance using an extension mechanism similar to the definition of new UCDs in the IVOA process, following the feedback from implementations of ObsTAP services.

The following characteristics are in common for all four attributes:

Attribute	Value
Datatype	adql:VARCHAR / Enum string
Units	NULL
UCD	meta.code.qual
Mandatory	0
Index	0 (except for o_calib_status, where it's "TBD")
Std	1

expand Telescope-Instrument model

From HST Archive coordination meeting: add a detector name.

Extend Observation Intent Type

STScI has been ingesting observations encapsulating images created by the Office of Public Outreach into our CAOM database. We've needed to use intent_type=science for these, but ideally we would prefer an outreach intent for them.
Partially implemented via PR opencadc/caom2tools#171 but opening an issue here for further discussion.

python module validation of observation_id field - too stringent?

Several of our collections at IRSA have observationid values that include spaces or slashes, and we're running into the following error trying to export them with the python caom2 library:

invalid observation_id: may not contain space ( ), slash (/), escape (\), or percent (%)

I reached out to @pdowler, who said this is to ensure that observation_id does not contain any characters aren't URI-safe, and suggested I open an issue here for discussion.

We don't use that field for URI population and haven't been able to find any documentation indicating that the observation_id string has any restrictions.

Is the caom2 module possibly being too stringent in how it validates this column?

(FYI we use CAOM 2.3, and the python module caom2 2.3.8.4)

add reference URL to proposal

to refer to the details of the propsal (abstract, document, etc)
some telescopes provide the endpoint

Observation.observationID and Plane.productID are too restrictive

In the code (java and python) these strings are restricted to being "valid path components".

These fields are used to generate several URIs:

ObservationURI of the form caom:{collection}/{observationID} for use as a reference in DerivedObservation.members

PlaneURI of the form caom:{collection}/{observationID}/{productID} for use as a reference in Plane.provenance.inputs

It is likely that Plane.creatorID is also assigned by using these values.

Although not part of the model, the CADC implementation (at least) uses these fields to generate Plane.publisherID that is the primary external reference to a Plane (product) and used as the input ID by the caom DataLink service.

restrict Algorithm.name values for simple observations

There are some defacto standard algorithm names in use (exposure) and a few others that could be restricted for use with simple observations only (simulation was the one that prompted allowing other names in the first place.

expand fields to allow multiple values

this comes from an archive partners slack discussion started by David Rodrigues

proposal.id
telescope.name
instrument.name
plane.energy.bandpassName

... maybe more

make PolarizationState more flexible

ObsCore and FITS WCS allow a limited set of states

actual data files often use combinations that better describe the polarization axis (even if degenerate aka 1 pixel)

CAOM-2.4 allows an extended set of values from https://listmgr.nrao.edu/pipermail/fitswcs/2008-March/thread.html

Recent metadata requests include additional things like Q + iU...

Add Artifact description lookup

We think it would be useful to have an element in the Artifact section that perhaps stores a URI that serves as a lookup for description information. MAST uses an internal ID to point to a table with descriptions for each type of file, which we pull for our interfaces. Storing this in the CAOM model itself can allow us to expand that so that we can share file descriptions across archives.

As an example, our table contains entries like these (plus internal IDs and other information):

DADS C3T file - Calibrated data WFPC2
exposure (L2c): 2D Calibrated data averaged over integrations
Calibrated full frame image

Provenance from multiple versions

This sort of relates to Issue opencadc/caom2#66 and the question of cardinality.

Planes are often produced from an ensemble of software, not a single application. In the case of ALMA MS data, in particular, we have MS data that is calibrated using CASA XXX and then split using CASA YYY. The provenance of CASA YYY would tell you that you should use YYY to open these files (MS is not a standard format) but the CASA XXX part is needed to tell you what the calibration system was. In particular CASA XXX is what tells you about calibration trust while CASA YYY part is more about data form. How (if at all) should this be expressed in the provenance?

align Plane metadata with IVOA Radio IG work on an ObsCore extention for radio

add
Observation.telescope.trackingMode : vocabulary

Plane.position.minBounds : shape
Plane.position.maxAngularScale : interval -- min/max scale
Plane.energy.resolution : double -- absolute resolution
Plane.energy.resolutionBounds : interval -- min/max abs res.
Plane.time.exposureBounds : interval -- min/max exposure time

Plane.uv : Visibility
distance : interval -- min/max
distributionEccentricity : double
distributionFill : double

** maybe add**
Plane.position.energyDependent : boolean -- range of shape, resolution, and scale is energy dependent

reconcile IVOA DALI xtype usage with Plane metadata

this is mainly about Shape and Multipolygon which are in WD-DALI-1.2 but there is still some disagreement between DALI authors.

also, in radio a simple observation is often characterised by a circle (maybe ellipse), so it is plausible that a DerivedObservation that combines data for larger sky coverage would have something like multi-circle or multi-shape.... TBD