njcuk9999 / lbl Goto Github PK
View Code? Open in Web Editor NEWLine by line code for radial velocity
License: MIT License
Line by line code for radial velocity
License: MIT License
If the lblrv directory does not exist when running compil, the code should have a more explicit error
Hello, I miss in the README the instructions for generating the drift file
cheers,
Andres
@eartigau no one apart from you has access to the list of files that generate the Mdwarf_temp_gradient.fits
file.
I would suggest you add another extension to Mdwarf_temp_gradient.fits
(a bin table)
You could have the filename, which instrument its from, the version of the pipeline etc just so anyone who has that files knows where it has come from.
There is currently no verification when using NIRPS-HA and NIRPS-HE instrumental setup that the S2D are actually from these modes. I successfuly used LBL on HA data using NIRPS-HE (by accident). We could add a assert looking at the "HIERARCH ESO INS MODE" keyword to make sure we are using LBL correctly
Same issue that was brought up before (apero-drs issue #752) for the apero_lbl_compile_nirps_ha recipe and reported as fixed. On the day of first noticing this error again (2024-06-26), there were 4 instances of the error. Here is an example of the error message:
class apero.core.core.drs_exceptions.LogExit : LBL Compile Exception [GL666B_GL666B] class lbl.core.base_classes.LblException : Unexpected lbl_compile.py error: class ValueError : zero-size array to reduction operation minimum which has no identity LBL Compile Exception [GL666B_GL514] class lbl.core.base_classes.LblException : Unexpected lbl_compile.py error: class ValueError : zero-size array to reduction operation minimum which has no identity ||
Here is a longer version of the error messages taken from the log file:
01:20:27.718-**|LBLCOMPILE_SCI[00179]|7 DTEMP files found.
01:20:27.727-**|LBLCOMPILE_SCI[00179]|Running LBL compile for GL666B_GL666B
01:20:34.378-**|LBLCOMPILE_SCI[00179]|Running LBL compile for GL666B_GL514
01:20:37.277-**|LBLCOMPILE_SCI[00179]|I[40-005-10001]: QUALITY CONTROL SUCCESSFUL - Well Done -
01:20:37.349-!!|LBLCOMPILE_SCI[00179]|LBL Compile Exception [GL666B_GL666B] <class 'lbl.core.base_classes.LblException'>: Unexpected lbl_compile.py error: <class 'ValueError'>: zero-size array to reduction operation minimum which has no identity
01:20:37.350-!!|LBLCOMPILE_SCI[00179]|
01:20:37.351-!!|LBLCOMPILE_SCI[00179]|LBL Compile Exception [GL666B_GL514] <class 'lbl.core.base_classes.LblException'>: Unexpected lbl_compile.py error: <class 'ValueError'>: zero-size array to reduction operation minimum which has no identity
01:20:37.361-!!|LBLCOMPILE_SCI[00179]|Traceback (most recent call last):
01:20:37.362-!!|LBLCOMPILE_SCI[00179]| File "/cosmos99/nirps/git-bin/apero-drs-online/apero/core/utils/drs_startup.py", line 433, in run
01:20:37.363-!!|LBLCOMPILE_SCI[00179]| llmain = func(recipe, params)
01:20:37.364-!!|LBLCOMPILE_SCI[00179]| File "/cosmos99/nirps/git-bin/apero-drs-online/apero/recipes/nirps_ha/apero_lbl_compile_nirps_ha.py", line 231, in __main__
01:20:37.365-!!|LBLCOMPILE_SCI[00179]| WLOG(params, 'error', '\n\n'.join(errors))
01:20:37.366-!!|LBLCOMPILE_SCI[00179]| File "/cosmos99/nirps/git-bin/apero-drs-online/apero/core/core/drs_log.py", line 428, in __call__
01:20:37.367-!!|LBLCOMPILE_SCI[00179]| raise drs_exceptions.LogExit(errorstring)
01:20:37.368-!!|LBLCOMPILE_SCI[00179]|apero.core.core.drs_exceptions.LogExit: LBL Compile Exception [GL666B_GL666B] <class 'lbl.core.base_classes.LblException'>: Unexpected lbl_compile.py error: <class 'ValueError'>: zero-size array to reduction operation minimum which has no identity
01:20:37.368-!!|LBLCOMPILE_SCI[00179]|
01:20:37.369-!!|LBLCOMPILE_SCI[00179]|LBL Compile Exception [GL666B_GL514] <class 'lbl.core.base_classes.LblException'>: Unexpected lbl_compile.py error: <class 'ValueError'>: zero-size array to reduction operation minimum which has no identity
01:20:37.370-!!|LBLCOMPILE_SCI[00179]|
01:20:37.371-!!|LBLCOMPILE_SCI[00179]|
01:20:37.384-**|LBLCOMPILE_SCI[00179]| ***************************************************************************
01:20:37.393-@!|LBLCOMPILE_SCI[00179]|W[40-003-00005]: Recipe apero_lbl_compile_nirps_ha has NOT been successfully completed
01:20:37.402-**|LBLCOMPILE_SCI[00179]| ***************************************************************************
I get a timeout error when running the NIRPS_ESO_demo at the point when it downloads the "model wave file" from the Goettinggen ftp:
2023-11-27 11:26:41.370|G| Downloading model wave file.
ftp://phoenix.astro.physik.uni-goettingen.de/HiResFITS/WAVE_PHOENIX-ACES-AGSS-COND-2011.fits
I checked the ftp address and it seems to be missing a "/data/" before the "/HiResFITS", so the complete URL should be:
http://phoenix.astro.physik.uni-goettingen.de/data/HiResFITS/WAVE_PHOENIX-ACES-AGSS-COND-2011.fits
Could you please check and update the code accordingly?
Thanks a lot!
Add LARGE_RV flag to wrapper / compute to allow a CCF to be run inside.
Flag would be True or False (false by default)
Flag is True does a CCF (apero style) inside LBL compute to give a rough start for an RV for each observation.
This is only for use for targets with a large RV signal (which means sysvel changes too much to use current approach)
test_wrap.py
Following a discussion with @eartigau last week, we determined that after the telluric pre-cleaning step the appropriate telluric corrected files are produced, however the RV code is instead using the uncorrected files.
Currently on the todo list:
Currently we support the original mode but should also support the ESO mode
test_data.py
The lbl/resources
folder doesn't get included in a pip install, resulting in
ModuleNotFoundError: No module named 'lbl.resources'
This can be easily fixed by adding a (empty) __init__.py
file in that folder.
Add data from Ryan Cloutier to LBL
The CARMENES wavelength solution provided by the CARACAL pipeline (the one used to reduce the data at the observatory by default) provides the wavelength solution in vacuum, not is air rest frame. However, according to the function defining how the CARMENES wavelength solution is read in the lbl
code carmenes.py
, the wavelength is assumed to be in air rest frame. The only correction applied is going from Angstroms to nm. Instead, it should first transform the wavelengths to air rest frame as indicated by the CARMENES pipeline builders through the following function:
ss = 1.e4/w
nn = 1 + 0.0000834254 + 0.02406147 / (130 - ss**2) + 0.00015998 / (38.9 - ss**2)
w = w/nn
Could you please implement this in the carmenes.py
module of lbl
?
Thanks.
I started looking at the code recently and have two short questions/remarks on the odd_ratio_mean
function:
2e-3
, but the default value value is 2e-4
. Is this intended or should the default value be the recommended one ?nmax
iterations would be useless (I have not run the lbl code on a significant dataset, so I don't know if this would have any impact in the the end, but intuitively it could sometimes help, a warning could also be raised if desired precision is not achieved after nmax
iterations).What are these files and how are they generated? Should they be regenerated when new templates are used?
(SPIRou) The exact BJD time between _AB and _C files differ inside lbl_OBJECT_TEMPLATE.rdb and drift.rdb files which insert Nan in the drift corrected version.
Bug found by @clairem789 which can be solved by changing line 2754 in lbl/science/general.py
to:
file_mask = np.abs(timestamp[row] - drift_table['rjd']) < 0.0007 # less than 1 minute
We need to decide if we apply this fix or we try to understand why the BJD time differ in the first place.
I got the following error when running the installation script:
...
Building wheels for collected packages: bottleneck
Building wheel for bottleneck (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /Users/clairemoutou/anaconda3/envs/lbl-env/bin/python /Users/clairemoutou/anaconda3/envs/lbl-env/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/03/70tg67hj12ggjmcwh0kr8jrr0000gn/T/tmpj0c1x8zl
ERROR: Failed building wheel for bottleneck
Failed to build bottleneck
ERROR: Could not build wheels for bottleneck which use PEP 517 and cannot be installed directly
Any clue?
I am trying to run LBL on an LFC file. I managed to 1) make a template which looks nice, 2) make a pos mask which nicely finds the peaks, 3) extract a few e2ds files and put them in the LBL tree.
Then I created a yaml configuration file, and ran lbl_compute.py --config=config_LFC.yaml
This gives an error; do you have an idea where it could come from, or how I could diagnose the problem?
thanks!
This is the message I got:
2021-06-01 17:10:51.953|G| log directory exists (Path=/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/log)
2021-06-01 17:10:52.036|I|
2021-06-01 17:10:52.036|I| *******************************************************************************
2021-06-01 17:10:52.036|I| LBL Compute
2021-06-01 17:10:52.036|I| VERSION: 0.0.011
2021-06-01 17:10:52.037|I| INSTRUMENT: SPIROU
2021-06-01 17:10:52.037|I| *******************************************************************************
2021-06-01 17:10:52.037|I|
2021-06-01 17:10:52.037|I| Command line arguments:
2021-06-01 17:10:52.037|I| --config="config_LFC.yaml"
2021-06-01 17:10:52.038|G| Mask directory exists (Path=/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/masks)
2021-06-01 17:10:52.038|G| Templates directory exists (Path=/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/templates)
2021-06-01 17:10:52.039|G| Calib directory exists (Path=/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/calib)
2021-06-01 17:10:52.039|G| Science directory exists (Path=/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/science)
2021-06-01 17:10:52.039|G| LBL RV directory exists (Path=/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/lblrv/LFC_LFC)
2021-06-01 17:10:52.040|G| LBL reftable directory exists (Path=/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/lblreftable)
2021-06-01 17:10:52.040|G| LBL rdb directory exists (Path=/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/lblrdb)
2021-06-01 17:10:52.041|G| Plot directory exists (Path=/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/plots)
2021-06-01 17:10:52.047|G| Science object directory exists (Path=/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/science/LFC)
2021-06-01 17:10:52.192|G| Defining all the template splines required later
2021-06-01 17:10:53.458|G| Loading bad odometer codes from spreadsheet
2021-06-01 17:11:24.718|G| 36078 bad values loaded
2021-06-01 17:11:24.718|I| *******************************************************************************
2021-06-01 17:11:24.719|I| Processing file 1 / 8 (7 left)
2021-06-01 17:11:24.719|I| *******************************************************************************
2021-06-01 17:11:24.773|G| SNR > 10 (SNR = 693.528000388146), passed SNR criteria
2021-06-01 17:11:25.921|G| CCF velocity (science) = 78.47 m/s
2021-06-01 17:11:30.494|G| CCF e-width = 1824.36 m/s
2021-06-01 17:11:30.496|G| --------------------------------------------------
2021-06-01 17:11:30.496|G| Iteration 1
2021-06-01 17:11:30.496|G| --------------------------------------------------
/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/lbl/science/general.py:749: RuntimeWarning: invalid value encountered in greater
sigmask = np.abs(nsig) > rms_sigclip_thres
2021-06-01 17:11:31.564|G| CCF velocity (model) = 147.99 m/s
2021-06-01 17:11:31.564|G| Model velocity 69.51 m/s
2021-06-01 17:11:31.565|G| --------------------------------------------------
2021-06-01 17:11:31.565|G| Iteration 2
2021-06-01 17:11:31.565|G| --------------------------------------------------
/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/lbl/science/general.py:749: RuntimeWarning: invalid value encountered in greater
sigmask = np.abs(nsig) > rms_sigclip_thres
2021-06-01 17:11:37.706|G| stdev_meas/stdev_pred = 1.28
2021-06-01 17:11:41.041|E| Unexpected lbl_compute error: <class 'numpy.core._exceptions.UFuncTypeError'>: ufunc 'subtract' did not contain a loop with signature matching types (dtype('<U3000'), dtype('<U3000')) -> dtype('<U3000')
Traceback (most recent call last):
File "/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/lbl/recipes/lbl_compute.py", line 80, in main
namespace = main(inst)
File "/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/lbl/recipes/lbl_compute.py", line 264, in main
model_velocity=model_velocity)
File "/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/lbl/science/general.py", line 916, in compute_rv
rv_final = np.array(dv + sys_rv - berv)
numpy.core._exceptions.UFuncTypeError: ufunc 'subtract' did not contain a loop with signature matching types (dtype('<U3000'), dtype('<U3000')) -> dtype('<U3000')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/lbl/recipes/lbl_compute.py", line 299, in
ll = main()
File "/net/GSP/nas12c/big_spirou/FULL_REDUCTION_CALIBDB_06131/lbl/lbl/recipes/lbl_compute.py", line 86, in main
raise LblException(emsg.format(*eargs))
lbl.core.base_classes.LblException: Unexpected lbl_compute error: <class 'numpy.core._exceptions.UFuncTypeError'>: ufunc 'subtract' did not contain a loop with signature matching types (dtype('<U3000'), dtype('<U3000')) -> dtype('<U3000')
When building the template of all _C.fits files, the majority of files are ignored
"SPIRou Object is classified as science. Found 473 files,ignoring 46415 other files for all FPs"
Has to do with the filter_files function in spirou.py
Currently on the TODO list:
spirou_cadc
class to test_wrap.py
test_wrap.py
lbl_seq is not yet documented
Hello,
I am running into CurveFit exceptions with stars of broad CCF. The exception also occurs when I'm using another star as template, which looks weird. Also, it happens now while it was not an issue before on the same stars. Also, sometimes the exception does not kill the LBL and I get the table at the end, but in one occurrence (example below) it does crash the job and produces no new file, and no rdb table. Any clue? Anyone having this too?
Thanks!
This is an example with FUORI:
Traceback (most recent call last):
File "/Users/moutou/miniconda3/envs/ap288/lib/python3.9/site-packages/lbl/recipes/lbl_compute.py", line 81, in main
namespace = main(inst)
File "/Users/moutou/miniconda3/envs/ap288/lib/python3.9/site-packages/lbl/recipes/lbl_compute.py", line 159, in main
systemic_vel_props = general.get_systemic_vel_props(inst, template_file,
File "/Users/moutou/miniconda3/envs/ap288/lib/python3.9/site-packages/lbl/science/general.py", line 550, in get_systemic_vel_props
raise LblException(emsg)
lbl.core.base_classes.LblException: CurveFit exception in rough CCF
Iteration = 1
P0 = [-1.92436321e+04 4.21759601e+04 2.08085222e+01 -1.84596027e+00
3.42332460e+00]
Function = science.general.py.get_systemic_vel_props()
Error: <class 'RuntimeError'>: Optimal parameters not found: Number of calls to function has reached maxfev = 1200.
I just noticed that, although the lbl*rdb files have a column name "rjd" for the date, the content is close to the MJD keywords in the lbl.fits files. rjd is expected to be BJD-2400000. So there is a ~0.5 day difference between MJD and rjd. I asked Pascal, who should have seen a problem when fitting the literature values for planets in DACE, but for some reason it doesn't seem to be an issue. So I'm confused. The lbl code says all over "MJD" so I guess this date in the rdb files really corresponds to some of the MJD keywords, but I actually don't seem to find which one. One example:
in lbl_GL411_GL411.rdb file 2374981o has "rjd" = 58528.62197768781
while dfits ../lblrv/GL411_GL411/2374981o*|grep JD has no corresponing value (about 600s apart from MJDATE):
MJDATE = 58528.6153547 / Modified Julian Date at start of observation
MJD-OBS = 58528.6153547 / Modified Julian Date at start of observation
MJDEND = 58528.6164779 / Modified Julian Date at end of observation
TCSMJD = 58528.6152604 / TCS MJD
MJDMID = 58528.61612320672 / Mid Observation time [mjd]
BJD = 2458529.121977688 / Barycentric Julian data calc. in BERVSRCE
Can you please clarify?
Sorry if this is a known issue.
When constructing the LBL ref table with HARPS-ESO data, the order numbering is offset by +1 compared to what it should be in the csv table. You get orders that go to 71 when then are 71 orders (the last one in python numbering should be 70). This causes a crash when running the code. I manually read the table, offset the order by -1 and saved the file and everything works now. This is not a problem at the compute level but at the time of ref file creation.
Hello, I have no clue why the drift file is not compiled from the LBL after a couple of tries. Many (49200) lblrv/FP_FP files have been computed and are present, and my lbl_run.ini file looks like this:
RUN_LBLREF = False
RUN_LBLMASK_FP = False
RUN_LBLCOMPUTE_FP = False
RUN_LBLCOMPILE_FP = True
RUN_LBLMASK_SCI = False
RUN_LBLCOMPUTE_SCI = False
RUN_LBLCOMPILE_SCI = False
SKIP_LBLREF = True
SKIP_LBLMASK_FP = True
SKIP_LBLCOMPUTE_FP = True
SKIP_LBLCOMPILE_FP = False
SKIP_LBLMASK_SCI = True
SKIP_LBLCOMPUTE_SCI = True
SKIP_LBLCOMPILE_SCI = True
I got this error:
2023-10-30 13:16:58.806|I| Producing LBL RDB 1 table
0%| | 4/49200 [00:02<8:40:39, 1.57it/s]
2023-10-30 13:17:01.347|E| Unexpected lbl_compile.py error: <class 'IndexError'>: boolean index did not match indexed array along dimension 0; dimension is 26423 but corresponding boolean dimension is 25400
12:17:01.364-**|LBLCOMPILE_FP[00000]| QUALITY CONTROL SUCCESSFUL - Well Done -
What can I do?
thanks!
Has GL411 and GL905 as OBJECT_TEMPLATE names, should be changed to GJ411 and GJ905 I guess, as OBJDRSN is used in the template names.
For the FP it is worth parallelising inside LBL
For individual objects it is worth leaving to the user / apero
From @clairem789 on njcuk9999/apero-drs#721
I tried running LBL on LFC files, AB and C separately.
The template looks good.
The trumpet plot looks like this: no RV values between 1300nm and 2200nm, where the flux is the best. sdv values exist and are of good quality for this range. What could have gone wrong?
For NIRPS spectra reduced with the ESO DRS, the header keyword for the blaze file is currently
HIERARCH ESO PRO REC1 CAL24 NAME
(in lbl/instruments/nirps.py
)
but sometimes it is actually found in
HIERARCH ESO PRO REC1 CAL25 NAME
Is there a way currently implemented to try both keywords?
From larnoldgithub on the apero-drs issues (njcuk9999/apero-drs#736)
bug or is just a key missing in lbl_run.in to select the instrument ?
2023-11-14 14:29:07.917|G|LBLMASK_FP[00001]| Downloading Mdwarf Temperature Gradient Table file
from: https://www.astro.umontreal.ca/~artigau/lbl/models/Mdwarf_temp_gradient.fits
to /data/spirou4/apero-data/offline/lbl/models/Mdwarf_temp_gradient.fits
2023-11-14 14:29:08.611|G|LBLMASK_FP[00001]| Skipped Mdwarf Temperature Gradient Table file. Missing from server.
2023-11-14 14:29:08.615|G|LBLMASK_FP[00001]| Downloading Mdwarf Mask [HARPS] file
from: https://www.astro.umontreal.ca/~artigau/lbl/models/mdwarf_harps.fits
to /data/spirou4/apero-data/offline/lbl/models/mdwarf_harps.fits
100% [..........................................................................] 1630080 / 16300802023-11-14 14:29:09.865|G|LBLMASK_FP[00001]|
Downloaded tapas file.
2023-11-14 14:29:09.866|G|LBLMASK_FP[00001]| Downloading Mdwarf Mask [NIRPS-HA] file
from: https://www.astro.umontreal.ca/~artigau/lbl/models/mdwarf_nirps_ha.fits
to /data/spirou4/apero-data/offline/lbl/models/mdwarf_nirps_ha.fits
100% [............................................................................] 921600 / 9216002023-11-14 14:29:11.096|G|LBLMASK_FP[00001]|
Downloaded tapas file.
2023-11-14 14:29:11.097|G|LBLMASK_FP[00001]| Downloading Mdwarf Mask [NIRPS-HE] file
from: https://www.astro.umontreal.ca/~artigau/lbl/models/mdwarf_nirps_he.fits
to /data/spirou4/apero-data/offline/lbl/models/mdwarf_nirps_he.fits
100% [............................................................................] 921600 / 9216002023-11-14 14:29:12.335|G|LBLMASK_FP[00001]|
Downloaded tapas file.
2023-11-14 14:29:12.336|G|LBLMASK_FP[00001]| Downloading Mdwarf Mask [SPIROU] file
from: https://www.astro.umontreal.ca/~artigau/lbl/models/mdwarf_spirou.fits
to /data/spirou4/apero-data/offline/lbl/models/mdwarf_spirou.fits
100% [............................................................................] 921600 / 9216002023-11-14 14:29:13.569|G|LBLMASK_FP[00001]|
Downloaded tapas file.
2023-11-14 14:29:13.570|G|LBLMASK_FP[00001]| Downloading Tapas file file
from: https://www.astro.umontreal.ca/~artigau/lbl/models/tapas_lbl.fits
to /data/spirou4/apero-data/offline/lbl/models/tapas_lbl.fits
100% [......................................................................] 180535680 / 1805356802023-11-14 14:29:23.443|G|LBLMASK_FP[00001]|
Downloaded tapas file.
The KW_TAU_OTHERS and the KW_NITERATIONS columns are mixed into a single column with NaN everywhere for ESPRESSO data sets.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.