jmmc-opendev/oitools

OiTools is a Java library dedicated to reading / writing OIFITS files

OITools

This is a java library dedicated to reading / writing OIFITS files (based on the nom.tam.fits library) developed by the JMMC technical team.

See Developers corner

OIFits standard versions:

• OIFITS V1
• OIFITS V2

More detailed data model including dynamic columns:

• OIFITS V1 - all
• OIFITS V2 - all

Setup

1. Download the last jar (oitools-TRUNK.jar) from release place https://github.com/JMMC-OpenDev/OITools/releases/

2. Call:

   • type in shell:

     java -cp _path_/_to_/oitools-TRUNK-jar-with-dependencies.jar fr.jmmc.oitools.OIFitsProcessor <parameters>

   • HINT: you can define aliases in your (bashrc) as of:

     alias oip="java -cp _path_/_to_/oitools-TRUNK-jar-with-dependencies.jar fr.jmmc.oitools.OIFitsProcessor"
     alias oilist="oip list"
     alias oiconvert="oip convert"
     alias oimerge="oip merge"

Usage & Examples

Examples are taken from the test folder

To get the command help, just type oip:

--------------------------------------------------------------------------------------
Usage: fr.jmmc.oitools.OIFitsProcessor command -o <path_output_file> <file locations>
------------- Arguments help ---------------------------------------------------------
| Key          Value           Description                                           |
|------------------------------------------------------------------------------------|
| command      help           Show this help                                         |
| command      list           List content of several oifits files                   |
| command      list_baselines List baselines and triplets used by several oifits files |
| command      dump           Dump the given oifits files                            |
| command      convert        Convert the given input file                           |
| command      merge          Merge several oifits files                             |
|------------------------------------------------------------------------------------|
| [-l] or [-log]              Enable logging (quiet by default)                      |
| [-v] or [-verbose]          Enable logging (quiet by default)                      |
| [-c] or [-check]            Check output file before writing                       |
| [-separation] <value>       Separation in arcsec for the target matcher            |
| [-o] or [-output] <file_path> Complete path, absolute or relative, for output file |
--------------------------------------------------------------------------------------
| Filter options available to the command merge:                     |
| [-target] <value>           Filter result on given Targets (comma-separated)       |
| [-insname] <value>          Filter result on given InsNames (comma-separated)      |
| [-night] <value>            Filter result on given Nights (integer, comma-separated) |
|                                                                                    |
| [-baselines] <values>       Filter result on given Baselines or Triplets (comma-separated) |
| [-mjds] <values>            Filter result on given MJD ranges (comma-separated pairs) |
| [-wavelengths] <values>     Filter result on given wavelength ranges (comma-separated pairs) |
|                                                                                    |
| <values> can be String or Range values (like 1e-6,2e-6) (comma-separated);         |
|          use the prefix 'not:' to have an exclusive filter                         |
|                                                                                    |
| Following columns may be available (OIFITS2 standard):                             |
| [-eff_wave] <values>     Filter result on given column values (comma-separated) |
| [-eff_band] <values>     Filter result on given column values (comma-separated) |
| [-sta_index] <values>     Filter result on given column values (comma-separated) |
| [-sta_conf] <values>     Filter result on given column values (comma-separated) |
| [-time] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-mjd] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-int_time] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-vis2data] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-vis2err] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-ucoord] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-vcoord] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-eff_wave] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-eff_band] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-hour_angle] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-radius] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-pos_angle] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-spatial_freq] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-u] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-v] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-snr_vis2] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-visamp] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-visamperr] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-visphi] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-visphierr] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-rvis] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-rviserr] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-ivis] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-iviserr] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-t3amp] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-t3amperr] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-t3phi] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-t3phierr] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-u1coord] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-v1coord] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-u2coord] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-v2coord] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-u1] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-v1] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-u2] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-v2] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-fluxdata] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-fluxerr] <values>     Filter result on given column ranges (comma-separated pairs) |
| [-snr_flux] <values>     Filter result on given column ranges (comma-separated pairs) |
--------------------------------------------------------------------------------------

• oilist input_file_location [input_file_location …] display the file content of input file(s) in CSV format files contents (granules like OiDB):

Example where two instruments are detected: the fringe tracker GRAVITY_FT with 5 spectral channels and the science detector GRAVITY_SC with 210 spectral channels

oilist GRAVI.2016-06-23T03:10:17.458_singlesciviscalibrated.fits
target_name     s_ra    s_dec   t_exptime       t_min   t_max   em_res_power    em_min  em_max  facility_name   instrument_name nb_vis  nb_vis2 nb_t3   nb_channels
IRAS17216-3801  261.2771541666667       -38.066788888888894     36.54999999999172       57562.13387079336       57562.134832901625      25.75830610631207       2.0264997147023678E-6   2.347295094295987E-6       VLTI    GRAVITY_FT      6       6       4       5
IRAS17216-3801  261.2771541666667       -38.066788888888894     0.0     57562.1339081338        57562.135323171286      1008.6521574069673      1.990000100704492E-6    2.4500000108673703E-6   VLTI    GRAVITY_SC 6       6       4       210

⚠ Warning: Targets are gathered according to their name and positions. Due to some uncertainties from files to files, the same target maybe duplicated. In this example, the target HD45677 is miss-identified as 3 different targets with varying right ascension.

 % oilist *.fits
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels
HD45677	97.071446	-13.05308	0.07500000298023224	9743.3964126119	9743.39698346838	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	0	12	8	64
HD45677	97.071446	-13.05308	0.07500000298023224	29230.186422152714	29230.186875347168	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	0	12	8	64
HD45677	97.071446	-13.05308	0.07500000298023224	58454.27455231501	58454.35846547426	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	72	72	48	64
HD45677	97.071446	-13.05308	0.07500000298023224	58455.17619694426	58455.2241775925	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	36	36	24	64
HD45677	97.071446	-13.05308	0.07500000298023224	58458.17127300924	58458.3521850002	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	126	186	124	64
HD45677	97.071446	-13.05308	0.07500000298023224	58459.1170581483	58459.32373210101	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	108	114	76	64
HD45677	97.071446	-13.05308	0.07500000298023224	58460.19688902766	58460.37100263876	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	48	72	48	64
HD45677_1	97.072017	-13.05308	0.07500000298023224	11692.253139791668	11692.271517135014	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	0	36	24	64
HD45677_1	97.072017	-13.05308	0.07500000298023224	58461.259056527546	58461.35039300915	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	36	36	24	64
HD45677_1	97.072017	-13.05308	0.07500000298023224	58464.21925374992	58464.296853532614	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	48	48	32	64
HD45677_1	97.072017	-13.05308	0.07500000298023224	58465.21247333342	58465.35976816349	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	90	96	64	64
HD45677_1	97.072017	-13.05308	0.07500000298023224	58466.225046249856	58466.2779251908	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	48	48	32	64
HD45677_2	97.072587	-13.05308	0.07500000298023224	58462.2034310185	58462.37512025083	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	114	120	80	64
HD45677_2	97.072587	-13.05308	0.07500000298023224	58463.31136550928	58463.36339001964	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	30	48	32	64

This can be fixed by ajusting the separation of the target matcher using the keyword -separation <value> (default: 1 arcsec ):

% oilist -separation 10  *.fits
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels
HD45677	97.071446	-13.05308	0.07500000298023224	9743.3964126119	9743.39698346838	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	0	12	8	64
HD45677	97.071446	-13.05308	0.07500000298023224	11692.253139791668	11692.271517135014	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	0	36	24	64
HD45677	97.071446	-13.05308	0.07500000298023224	29230.186422152714	29230.186875347168	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	0	12	8	64
HD45677	97.071446	-13.05308	0.07500000298023224	58454.27455231501	58454.35846547426	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	72	72	48	64
HD45677	97.071446	-13.05308	0.07500000298023224	58455.17619694426	58455.2241775925	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	36	36	24	64
HD45677	97.071446	-13.05308	0.07500000298023224	58458.17127300924	58458.3521850002	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	126	186	124	64
HD45677	97.071446	-13.05308	0.07500000298023224	58459.1170581483	58459.32373210101	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	108	114	76	64
HD45677	97.071446	-13.05308	0.07500000298023224	58460.19688902766	58460.37100263876	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	48	72	48	64
HD45677	97.071446	-13.05308	0.07500000298023224	58461.259056527546	58461.35039300915	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	36	36	24	64
HD45677	97.071446	-13.05308	0.07500000298023224	58462.2034310185	58462.37512025083	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	114	120	80	64
HD45677	97.071446	-13.05308	0.07500000298023224	58463.31136550928	58463.36339001964	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	30	48	32	64
HD45677	97.071446	-13.05308	0.07500000298023224	58464.21925374992	58464.296853532614	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	48	48	32	64
HD45677	97.071446	-13.05308	0.07500000298023224	58465.21247333342	58465.35976816349	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	90	96	64	64
HD45677	97.071446	-13.05308	0.07500000298023224	58466.225046249856	58466.2779251908	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	48	48	32	64

• oip list_baselines display similar information as oilist plus the station/baselines used.

Example showing that the stations used are: A0 B2 C1 D0 and the file contains all combinaison of baselines and triplets:

oip list_baselines GRAVI.2016-06-23T03:10:17.458_singlesciviscalibrated.fits
instrument_name	em_min	em_max	night_id	target_name	s_ra	s_dec	mjds	baselines
GRAVITY_FT	2.0264997147023678E-6	2.347295094295987E-6	57562	IRAS17216-3801	261.2771541666667	-38.066788888888894	[57562.133866,57562.136111] [57562.133947,57562.136748] [57562.13412,57562.13522] [57562.13412,57562.135231] [57562.13412,57562.135243] [57562.134745,57562.135336] [57562.13478,57562.135243] [57562.13478,57562.135498] [57562.134815,57562.135266] [57562.134826,57562.135266] 	A0 B2 C1 D0 B2-A0 C1-A0 C1-B2 C1-D0 D0-A0 D0-B2 C1-B2-A0 C1-D0-A0 C1-D0-B2 D0-B2-A0
GRAVITY_SC	1.990000100704492E-6	2.4500000108673703E-6	57562	IRAS17216-3801	261.2771541666667	-38.066788888888894	[57562.13390,57562.136343] [57562.13397,57562.133981] [57562.13397,57562.137454] [57562.135312,57562.13544] 	A0 B2 C1 D0 B2-A0 C1-A0 C1-B2 C1-D0 D0-A0 D0-B2 C1-B2-A0 C1-D0-A0 C1-D0-B2 D0-B2-A0

• oip dump print headers of all HDUs.

• oimerge -o <output_file_path> input_file_location [input_file_location …] merge multiple files with filter support, targets are gathered and merged into a single OI_TARGET table.

  Available filters are (multiple arguments must be comma separated):

  • -target
  • -insname
  • -night
  • -mjds
  • -baselines
  • -wavelength

• Extracting flux from Gravity science instrument at wavelengths between 2.15 and 2.2 μm*:

% oimerge -insname GRAVITY_SC  -baselines A0,B2,C1,D0 -wavelengths 2.15E-6,2.20E-6 -o output.fits  GRAVI.2016-06-23T03:10:17.458_singlesciviscalibrated.fits
Writing: output.fits
% oilist output.fits
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels
IRAS17216-3801	261.2771541666667	-38.066788888888894	300.0	57562.133972453696	57562.133972453696	988.1521592210469	2.1506698431039695E-6	2.1990908862790093E-6	VLTI	GRAVITY_SC	0	0	0	23

OIFitsViewer

You may also define some other aliases to dump main informations of your oifits files through XML or TSV format or check content

 alias oiv="java -cp _path_/_to_/oitools-TRUNK-jar-with-dependencies.jar fr.jmmc.oitools.OIFitsViewer"
 alias oicheck="oiv -check"

To get the command help, just type oiv:

No file location given in arguments.
-------------------------------------------------------------------------
Usage: fr.jmmc.oitools.OIFitsViewer [-f|-format] [-v|-verbose] [-t|-tsv] <file locations>
------------- Arguments help --------------------------------------------
| Key          Value           Description                              |
|-----------------------------------------------------------------------|
| [-h|-help]                   Show arguments help                      |
| [-l] or [-log]               Enable logging (quiet by default)        |
| [-f] or [-format]            Use the number formatter                 |
| [-v] or [-verbose]           Dump all column data                     |
| [-t] or [-tsv]               Dump object table in tsv format          |
| [-c] or [-check]             Check only given file(s)                 |
| [-g] or [-granule]           Use Granules to get target metadata      |
-------------------------------------------------------------------------

oitools's Issues

OI_INSPOL table revision

1-In OIFitsLoader the method processHDUnits() read OIFits Tables in the order of the file. Table OI_INSPOL need information (INSNAME, ARRNAME, ...) contained in other tables.

The first proposal could be to add an if loop that browses the file a first time and records the priority tables. But that would still add a loop of reading of the file (there are already four).

2-The chicken-egg problems on nWaves: how to set nWaves as columns are loaded after the constructor ?
Actually there is a hack on nWaves (it works well except the validation)

Consistency for failures messages

Need to have messages about failures more uniform:
Order of HDU, MEMBER, VALUE, ROW or COL is always the same in all failures messages.

Refactor MJD filter

Is your feature request related to a problem? Please describe.
No problem using such filter using current CLI interface : this issue is a technical task

Main goal is to refactor current MJD filter in a simpler manner so it can be a starting example for next coming 1D filters.

Fix FitsImageLoader.readHDU()

Restore hack:

if (ImageHDU.isHeader(hdr)) {

•        // Hack for ImageHDU having NAXIS > 2 and NAXISn=1
  

•        fixAxesInHeader(hdr);
  

•    }
  

See ecd1edf

Unconsistancies in oilist output

In oilist the number of data: nb_vis, nb_vis2, nb_t3 and nb_channels seems to be missing and/or fused together as it can be seen in the output of these 3 commands (2 with a single files the last one with both):

oilist 2018-12-02T062655_HD45677_IR-LM_IN_IN_cal_oifits_0.fits                                                         
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels	
HD45677	97.071446	-13.05308	0.07500000298023224	58454.27455231501	58454.27875227183	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	664	

oilist 2018-12-02T062655_HD45677_IR-LM_IN_OUT_cal_oifits_0.fits 
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels	
HD45677	97.071446	-13.05308	0.07500000298023224	58454.27638495392	58454.27638495392	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	6	6	4	64	

oilist 2018-12-02T062655_HD45677_IR-LM_IN_IN_cal_oifits_0.fits 2018-12-02T062655_HD45677_IR-LM_IN_OUT_cal_oifits_0.fits 
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels	
HD45677	97.071446	-13.05308	0.07500000298023224	58454.27455231501	58454.27875227183	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	112	8	64	

Load and write extra Tables and columns

In OIFitsLoader, loader does not support Tables and columns that are not in the standard. it's throws the rule: OIFITS_TABLE_UNKNOWN or COL_UNKNOWN.
The ideal would be able to load these Tables and columns (even without analyzing them) so the users find their all information.

• Create metadata for the descriptors in the Tables.
• For columns create a column descriptor and add them.
• Create metadata and adding descriptors makes these unknown Tables and columns run automatically.

Always support complex column: VISDATA & VISERR (OI_VIS)

In OI_VIS, there can be optional columns for complex visibilities (on ASPRO or AMBER - not OIFits), VISDATA and VISERR.
For the moment, this is the method hasOiVisComplexSupport() who controls their presence. True will include optional columns in the datamodel, else will ignore them

• Always create the columns if they are present in the files ?
• Create JUnit test ?

Review the Rule message for UNIT_EXIST

Actual : "Missing unit for column '{{MEMBER}}', should be not empty or '{{VALUE}}'"

Merge method read() in FitsImageLoader and OIFitsLoader

We have two methods (read()) to read HDU:

In FitsImageLoader:
/** Return all image HDUs read from the given Fits object
used: read(fitsFile) call => List read(final Fits fitsFile){}

In OIFitsLoader call read() method in Fits (nom.tam.fits package):
/** Return all HDUs for the Fits object.
used: fitsFile.read() call => BasicHDU[] read(){}

In a first time, we need to manage the ticket linked to the nom.tam.fits library, maybe there has been some change on this method.
In a second time, in FitsImageLoader we have a method fixAxesInHeader() (Fix header for degenerated AXES (ie NAXISn = 1) ie remove such axes) that must be done before.
Finaly, we can identify the commonalities between the methods in order to identify a common part.

Expose/manage a version number

Even if we still continue to use TRUNK version in our mvn artifcats, such version number could be printed in the command lines output or xml reports.

Initialize keyword to default value

When the method: OIFitsChecker.isInspectRules() has been added in the code to catch failures, we can go into methods with null values.
To prevent this problem a number of checks (on values that can be null) have been created:

if (value == null) {
}

In this case on FitsImageLoader, we propose to initialize the values to a default value so that it can not be null anymore and ensure all mandatory keywords in memory have a not null value (getters should never return null)

Management of additional oifits columns/tables by OiFitsExplorer

DataModel JavaDoc and Wiki

Need documentation on creating the DataModel.
Added a clearer Javadoc for the DataModel code.
Why we load multiple files and why the creation of a memory structure.

Derived columns need to use method getMinMaxColumnValue()

Method getMinMaxColumnValue(final String name) return the minimum and maximum column value given its name. The returned value can be null if the column has never been defined.
This method is defined in FitsTable and used some getColumnAs() to find the values.

To improve the method, it should be rewritten to take into account getDerivedColumnAs().

OIFitsFile: make wlen min max getters more generic

The method computeWavelengthBounds() compute min and max wavelength. This method reflects the min and max values of every OI_WAVELENGTH tables.
By making this method more generic one could create statistic calculations on several values. Rewrite this method for a calculation of statistics on different tables.

Improve the method is3D()

Specific method: is3D() return true if the value is multiple (3D array).
the method: isArray() already exists, return true if the value is multiple (array).

Need to improve the method isArray() to take into account 3D and more.
Proposal:
Use switch on getRepeat() (dimensions of the column).

No error if DIM COL <= DIM MAX COL

Lenient Format. In ColumnMeta.java

Actual:
if (columnRepeat != descRepeat)

Proposal:
if (columnRepeat <= descRepeat)

Handle Severity Profiles

Describe the needs:
Severity profiles mark validation failures with the appropriate severity: INFO, WARNING, ERROR

Multiple instances of the GRAVITY_FT INSNAME

Description

While testing OifitsExplorer version 0.3.0 beta 10 on a collection of 20 oifits GRAVITY files of the same target I notice the following weakness. The TARGET windows shows multiple folders with names "GRAVITY_FT_#" with # going from 1 to 9 (despite the INSNAME in each table being always GRAVITY_FT, (ee screenshot below). Since the interface does not allow a practical multiple selection of folders this makes it impossible to read all the content of the GRAVITY_FT data at once simply.

screenshot: https://cloud.univ-grenoble-alpes.fr/s/wgbRGM34rws7ptH

Additionally, while looking at the FOLDER tooltips it seems that almost all folders correspond to the same number of channels, Lambda min, max and resolution. So this grouping by folders is not obvious to understand.

Desirable evolution

I currently don't see any real justification for separating the different folders like so. I would think that all GRAVITY_FT data should appear under the same folder name GRAVITY_FT.

extend validation rules with important primary header keywords

Rename OISpectrum class to OIFlux

Simple code refactoring

Facility name disapears with merging

When I merge GW_Ori files (files from LITpro test folder in CVS), the facility name becomes undefined:

% oilist GW_Ori.200*
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels	
GW_Ori	82.28334	11.87	0.0	52973.32806712948	52973.38412037026	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	IOTA_AB	IONIC3	0	18	6	1	
GW_Ori	82.28334	11.87	0.0	52974.37421296304	52974.37674768502	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	IOTA_AB	IONIC3	0	9	3	1	
GW_Ori	82.28334	11.87	0.0	53351.20174189797	53351.39668981498	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	IOTA_AB	IONIC3	0	57	19	1	
GW_Ori	82.28334	11.87	0.0	53353.34997685207	53353.38156828703	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	IOTA_AB	IONIC3	0	18	6	1	
GW_Ori	82.28334	11.87	0.0	53354.35648148134	53354.38167824084	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	IOTA_AB	IONIC3	0	12	4	1	
GW_Ori	82.28334	11.87	0.0	53355.22764467588	53355.26708333334	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	IOTA_AB	IONIC3	0	15	5	1	
GW_Ori	82.28334	11.87	0.0	53359.32408564817	53359.35291666677	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	IOTA_AB	IONIC3	0	9	3	1	
% oimerge -o GW_Ori.2004Dec_merged.oifits GW_Ori.2004Dec.A28B*
Writing: GW_Ori.2004Dec_merged.oifits
% oilist GW_Ori.2004Dec_merged.oifits 
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels	
GW_Ori	82.28334	11.87	0.0	53351.20174189797	53351.39668981498	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	UNDEFINED	IONIC3	0	57	19	1	
GW_Ori	82.28334	11.87	0.0	53353.34997685207	53353.38156828703	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	UNDEFINED	IONIC3	0	18	6	1	
GW_Ori	82.28334	11.87	0.0	53354.35648148134	53354.38167824084	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	UNDEFINED	IONIC3	0	12	4	1	
GW_Ori	82.28334	11.87	0.0	53355.22764467588	53355.26708333334	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	UNDEFINED	IONIC3	0	15	5	1	
GW_Ori	82.28334	11.87	0.0	53359.32408564817	53359.35291666677	6.6666665135530785	1.6499999446750735E-6	1.6499999446750735E-6	UNDEFINED	IONIC3	0	9	3	1	

Pretty print oilist

oip list and oip list_baselines output are a bit messy and it is difficult to assess which value belong to which field.
As an example:

oilist 2018-12-14T051749_HD45677_IR-LM_IN_IN_cal_oifits_0.fits
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels	
HD45677	97.072017	-13.05308	0.07500000298023224	58466.225046249856	58466.23150952865	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	6	6	4	64

Create Method to get the OIFITS standard of the OIFitsFile

FitsConstants.KEYWORD_CONTENT_OIFITS2.equals(FitsConstants.KEYWORD_CONTENT) ? OIFitsStandard.VERSION_2 : OIFitsStandard.VERSION_1) is used 4 time !

Proposal:
/**
* define the version of the file
*
* @return the version
*/
public OIFitsStandard defineOIFITSVersion() {
return (FitsConstants.KEYWORD_CONTENT_OIFITS2.equals(FitsConstants.KEYWORD_CONTENT) ? OIFitsStandard.VERSION_2 : OIFitsStandard.VERSION_1);
}

Junit Test loading lots of files when isInspectRule is active

Create a JUnit Test with isInspectRule = true that loads all FITS files.
Objective: test the code robustness

Implement OI_INSPOL_INSNAME_UNIQ Rule

The Rule: OI_INSPOL_INSNAME_UNIQ exist in Rule and was not implemented in the OI_INSPOL Table:
check if the INSNAME column values are only present in a single OI_INSPOL table (compare multi OI_INSPOL table)

Merge duplicated ruleFailed() method for the rule : OIFITS_TABLE_UNKNOWN

Currently we have two methods ruleFailed():
-ruleFailed(final Rule rule, final FitsHDU hdu, final String member)
-ruleFailed(final Rule rule, final String extName, final int extNb)

OIFITS_TABLE_UNKNOWN is the only rule that uses the second version of this method.
It is therefore a question of mutualising the code of these methods or of making sure that the OIFITS_TABLE_UNKNOWN rule can use the common method.

Error

OI_INSPOL missing in description in xml output

Check and correct the presence of OI_INSPOL in the xml output in XmlOutputVisitor

Fix extNb for FITS image HDU

Take into account the FitsImageHDU when counting the FITS HDU (extNb field)

Import initial code snapshot

Import code from our (jmmc) svn server BUT :

• add file header (GPL v3)
• check copyrights for all data / source files
• ensure synchronization between SVN <=> GIT

improve the code for inspectRuleFailed()

For some rules:

we use 2 method inspectRuleFailed() to catch rule and ruleFailed() to catch failures (and ruleFailed catch rules too) there is duplication of rules capture which creates several problems:

-We have some rules with no hdu information that is why applyTo and version is give by a hack:
Rule passed in the Hack:
HACK: inspectRuleFailed FILE for 'FILE_EXIST'
HACK: inspectRuleFailed FILE for 'FILE_LOAD'
HACK: inspectRuleFailed FILE for 'OIFITS_OI_ARRAY_EXIST_V2'
HACK: inspectRuleFailed FILE for 'OIFITS_OIDATA'
HACK: inspectRuleFailed FILE for 'OIFITS_OI_TARGET_EXIST'
HACK: inspectRuleFailed FILE for 'OIFITS_OI_WAVELENGTH_EXIST'
HACK: inspectRuleFailed FILE for 'OIFITS_PRIMARYHDU_EXIST_V2'

-This badly generated hack gives rise to unwanted applyTo duplication:
Rule have conflict in there applyTo:
BAD inspectRuleFailed applyTo conflict for 'OIFITS_OI_TARGET_EXIST': [OI_TARGET]
BAD inspectRuleFailed applyTo conflict for 'OIFITS_OI_WAVELENGTH_EXIST': [OI_WAVELENGTH]
BAD inspectRuleFailed applyTo conflict for 'OIFITS_PRIMARYHDU_EXIST_V2': [MAIN_HEADER]
BAD inspectRuleFailed applyTo conflict for 'OIFITS_OI_ARRAY_EXIST_V2': [OI_ARRAY]

Nom.tam.fits library update

The version of the library that we use has evolved:
-retrieve and compare versions of GitHub (the one we use and the last one)
-feasibility study: see at which level the changes made will impact our code.
-delete our library and add the latest version

Instantiate a test: Delete our version of Nom.tam.fits and put the new version in the code. Note all the impacted points.

Returned all HDUs information even if no image in this HDU

This issue concerns only images. In the code HDU with empty images are skip.
In the case of a file containing several HDUs, some of them without images, there could be a loss of information on the keywords of these HDUs.

In FitsImageLoader when the code run processHDUnits (when a file is load) if the hdu is an instance of ImageHDU, a FitsImageHDU is created. The code analyse the hdu, processKeyword, ...
But the code register the imageHDU only if the FitsImageHDU has images. The idea is to find a way to return imageHDU even if there are no image to hold keywords.

Find a way to reproduce the bug?
Pay attention to other places in the code that could impact.

Comparison with constants in OIFitsFile

We have in OiFitsFile a comparison with constants (used two times):

(FitsConstants.KEYWORD_CONTENT_OIFITS2.equals(FitsConstants.KEYWORD_CONTENT) ? OIFitsStandard.VERSION_2 : OIFitsStandard.VERSION_1)

We need to check its effect

Proposal:
Can delete the method : checker.inspectRuleFailed() (Special case which catch the failures using directly the applyTo and version information)

To

checker.ruleFailed(Rule.OIFITS_OI_TARGET_EXIST);
which now can process rules begin with OIFITS (And add "FILE" in the applyTo for this rules).

Draw UML diagrams and explanatory diagrams on the structure of the code

UML: Load files.
UML: Failures.
TODO...

NaN when separation is too large

target (ra,dec) become NaN when separation >= 180

oilist -separation 180 2018-12-02T062655_HD45677_IR-LM_IN_IN_cal_oifits_0.fits 
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels	
UNDEFINED	NaN	NaN	0.07500000298023224	58454.27455231501	58454.27875227183	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	6	6	4	64

oilist -separation 179 2018-12-02T062655_HD45677_IR-LM_IN_IN_cal_oifits_0.fits
target_name	s_ra	s_dec	t_exptime	t_min	t_max	em_res_power	em_min	em_max	facility_name	instrument_name	nb_vis	nb_vis2	nb_t3	nb_channels	
HD45677	97.071446	-13.05308	0.07500000298023224	58454.27455231501	58454.27875227183	47.39237355339858	2.841224613803206E-6	4.209076450933935E-6	VLTI	MATISSE	6	6	4	64