franpoz / sherlock Goto Github PK
View Code? Open in Web Editor NEWEasy and versatile open-source code to explore Kepler, K2 and TESS data in the search for exoplanets
License: MIT License
Easy and versatile open-source code to explore Kepler, K2 and TESS data in the search for exoplanets
License: MIT License
This would be useful for batch executions where the user doesn't know very well the content of the lightcurves.
Remove the binning plot in the detrending plots.
It is not offering to much information. It is better to let only the corrected un-binned plots
As well as we do a savgol filter with a short cadence, we could provide an open function as we do for ois filtering to do any processing wanted by the user before sherlock begins with the runs.
We could add a setup method which would allow the user to overwrite the SignalSelector interface and provide his implementation to sherlock.
Save the time, flux, flux_err after SHERLOCK operations in a csv file to proceed with vetting and fits steps after.
For a given signal that we want to vett and fit, we need as imputs its period and T0, which has been provided by SHERLOCK previously.
That way the pipeline could be executed from any path and it would always find the data and thus, it wouldn't need to download again.
We are generating a considerable amount of plots in the results. It is more practical to save independently each run of SHERLOCK in a folder.
Like habitable zones or optimistic zones, ice limit, tidal lock range etc
As LATTE plots are not very useful for shallow transits we might add our custom plots of the transits with the flux values limited by a depth value. That way we could focus the view into the values that we really want to explore.
We are using only the essential parameters for the fit. We can add some more like the limb darkening parameters and many others might not be not critical for the fit but could be helpful to increase its accuracy.
Validation is right now anecdotal and needs to be pushed at least at user input levels.
Right now we choose the lowest period from the strongest ones. It might be better to just choose the stronger one or think about something different.
One of the analysis points which are not solved by SHERLOCK yet is the visual processing of every detrend result for each run. We could add a new mechanism to reject false positives returned by TLS by analyzing the found transit environment. For instance, we could assess sinusoidal trends, high variability of signal, empty measurements around the transit...
Sometimes SHERLOCK leaves out of the final promising candidates list one or several signals that could also be relevant. Thus, planet radius and r_star/r_planet ratio would be useful when printed besides each found signal for each detrend.
In order to apply a proper fitting of the transits via [ALLESFITER] (#25)
We need to provide the ratio between the planet and the star. This can be easily given in the log files making use of the
results.rp_rs
line in TLS
We should include an option to run a vetting similar to how is done in LATTE
This way we avoid limiting for instance to 2 sectors which are too far in the time.
Example for TIC 299798795:
tls_results.folded_phase[np.argwhere(tls_results.folded_phase_model < 1)]
This returns a range between 0.495 and 0.505 for the period of 4.17 which would be a duration of 60 mins aprox. However, the tls returned duration is 13 minutes.
Possible bug for transitleastsquares.
When running the TIC 467179258, in the 3rd run is detected a signal which all the detected transits are in borders. However, the border score indicates that they are not (Border score=1).
This needs further exploration.
win_size Period Per_err N.Tran Mean Depth (ppt) T. dur (min) T0 SNR SDE FAP Border_score
PDCSAP_FLUX 32.01687 0.104577 1 1.899 138.0 1710.2785 8.614 24.781 8.0032e-05 1.00
Example:
vet.py --object_dir MIS_TIC_307210830_all --candidate 5
Traceback (most recent call last):
File "/home/martin/git_repositories/sherlockpipe/sherlockpipe/vet.py", line 162, in __process
dec, self.args)
File "/home/martin/git_repositories/sherlockpipe/watson/lib/python3.6/site-packages/LATTE/LATTEbrew.py", line 363, in brew_LATTE
ldv.LATTE_DV(tic, indir, syspath, transit_list, sectors_all, target_ra, target_dec, tessmag, teff, srad, [0], [0], tpf_corrupt, astroquery_corrupt, FFI = False, bls = False, model = model, mpi = args.mpi)
File "/home/martin/git_repositories/sherlockpipe/watson/lib/python3.6/site-packages/LATTE/LATTE_DV.py", line 89, in LATTE_DV
TOI = (float(TOIpl["Full TOI ID"]))
File "/home/martin/git_repositories/sherlockpipe/watson/lib/python3.6/site-packages/pandas/core/series.py", line 129, in wrapper
raise TypeError(f"cannot convert the series to {converter}")
TypeError: cannot convert the series to <class 'float'>
couldn't download TP but continue anyway
The first draft is already posted in the paper
folder. Needs further details and development.
The intention is to submit to https://joss.theoj.org
To run the vetting via LATTE
package, it should be used an old version of matplotlib
. We may use the same for the full sherlock
to avoid the need of a virtual env for the vetting part.
Armonics, subharmonics and the source signal can be detected by several detrends in the same run. Maybe we can enhance the signal scoring by detecting whether the found signal is armonic or subharmonic of some others which were found in the same run.
The current nomenclature for working folders is given as MIS_TIC XXXXX, which is no longer convenient.
Let's move to a new one as TICXXXXX_2MIN or TICXXXXXX_FFI etc.
It seems like a TLS issue, but I create it here until we confirm it. It happened in the first run of this execution:
sherlock = Sherlock([]).setup_detrend(initial_rms_threshold=2.5, initial_rms_bin_hours=5)\
.setup_transit_adjust_params(snr_min=8, period_min=0.5, period_max=70, max_runs=8)\
.load_ois().filter_hj_ois().limit_ois(7, 8).run()
The exception was:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/home/martin/.local/lib/python3.6/site-packages/transitleastsquares/core.py", line 153, in search_period
duration_min_in_samples = int(numpy.floor(duration_min * len(y)))
ValueError: cannot convert float NaN to integer
Give several examples of the usage of Sherlock through sherlock_user_properties.python
From MIS_TIC_XXXXXX_all to TICXXXXXX_2MIN_all
So the algorithm is open to new definitions given by any user without needing to push their custom solution into the main branch.
We need to update how to load yaml files. The current version yields this warning:
/usr/local/lib/python3.6/dist-packages/sherlockpipe/__main__.py:16: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
sherlock_user_properties = yaml.load(open(resources_dir + "/" + 'properties.yaml'))
/usr/local/lib/python3.6/dist-packages/sherlockpipe/__main__.py:17: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
user_properties = yaml.load(open(args.properties))
That way long executions which are interrupted can be resumed instead of restarted.
complete the list of installed packages needed with allesfitter and yaml
To provide with an in-depth vetting of a promising signal, we should consider adapt LATTE criteria.
tls_results.power represent the SDE vs period of every detrended signal. We could look for every found signal period into the otherdetrended signals power spectrum to see if it is also strong enough (even not being the stronghest for its source signal).
After a detection by SHERLOCK the solution should be improved by a proper fit to have the best ephemeris as possible to perform a ground-based follow-yp. We might use Allesfitter/Juliet/Exofast for that.
Right now auto-detrend only works for one period. It would be a great addition to do a detrend or subtraction of pulsations from pulsating stars. For this we can follow the asteroseismology examples from Lightkurve. In order to be able to do that we need to be able to automatically characterize the kind of object we are processing to select the proper algorithm to detrend it.
For fast rotators, auto-detrend should be done only for the strongest period and not for the shortest one within a range of strong periods.
The current version of the file that one needs for the fitting looks like:
# This example YAML file illustrates the provisioning of planet and star parameters. The star mass here is important
# because the semi-major axis is not being provided and thus, it'd need to be calculated from the period and the M_star.
settings:
cpus: 7
planet:
id: 231663901
period: 1.4309133565904666
t0: 1326.004206688168
rp_rs: 0.10455530183261129
star:
R_star: 1.02
M_star: 1.05
But we need to also use the errors given by TLS, at least in the period
. For the case of T0
, TLS does not provide any error, but I would say that we can use a reasonable value of +-0.01 days
This is useful not only for reviewing reports but also for subsequent future fitting.
To obtain the best solution as possible once we have the TLS results we need to fit the data properly. We choose
Allesfitter.
We will need three files:
1) photometic file: TESS.csv
it should be a coma separated csv file with three columns; #time, flux, flux_err
# time,flux,flux_err
0.000000000000000000e+00,1.000993428306022448e+00,2.000000000000000042e-03
1.041666666666666609e-02,9.997246288021938154e-01,2.000000000000000042e-03
2.083333333333333218e-02,1.001297691868046513e+00,2.000000000000000042e-03
2) settings.csv
This file can be constant for all our trials, in principle there is not need to change to many things. It looks like:
#name,value
###############################################################################,
# General settings,
###############################################################################,
companions_phot,b
companions_rv,
inst_phot,Leonardo
inst_rv,
###############################################################################,
# Fit performance settings,
###############################################################################,
multiprocess,True
multiprocess_cores,4
fast_fit,True
fast_fit_width,0.3333333333333333
secondary_eclipse,False
phase_curve,False
shift_epoch,True
inst_for_b_epoch,all
###############################################################################,
# MCMC settings,
###############################################################################,
mcmc_nwalkers,100
mcmc_total_steps,2000
mcmc_burn_steps,1000
mcmc_thin_by,1
###############################################################################,
# Nested Sampling settings,
###############################################################################,
ns_modus,dynamic
ns_nlive,500
ns_bound,single
ns_sample,rwalk
ns_tol,0.01
###############################################################################,
# Limb darkening law per object and instrument,
# if 'lin' one corresponding parameter called 'ldc_q1_inst' has to be given in params.csv,
# if 'quad' two corresponding parameter called 'ldc_q1_inst' and 'ldc_q2_inst' have to be given in params.csv,
# if 'sing' three corresponding parameter called 'ldc_q1_inst'; 'ldc_q2_inst' and 'ldc_q3_inst' have to be given in params.csv,
###############################################################################,
host_ld_law_Leonardo,quad
###############################################################################,
# Baseline settings per instrument,
# baseline params per instrument: sample_offset / sample_linear / sample_GP / hybrid_offset / hybrid_poly_1 / hybrid_poly_2 / hybrid_poly_3 / hybrid_pol_4 / hybrid_spline / hybrid_GP,
# if 'sample_offset' one corresponding parameter called 'baseline_offset_key_inst' has to be given in params.csv,
# if 'sample_linear' two corresponding parameters called 'baseline_a_key_inst' and 'baseline_b_key_inst' have to be given in params.csv,
# if 'sample_GP' two corresponding parameters called 'baseline_gp1_key_inst' and 'baseline_gp2_key_inst' have to be given in params.csv,
###############################################################################,
baseline_flux_Leonardo,hybrid_offset
###############################################################################,
# Error settings per instrument,
# errors (overall scaling) per instrument: sample / hybrid,
# if 'sample' one corresponding parameter called 'log_err_key_inst' (photometry) or 'log_jitter_key_inst' (RV) has to be given in params.csv,
###############################################################################,
error_flux_Leonardo,sample
###############################################################################,
# Exposure times for interpolation,
# needs to be in the same units as the time series,
# if not given the observing times will not be interpolated leading to biased results,
###############################################################################,
t_exp_Leonardo,
###############################################################################,
# Number of points for exposure interpolation,
# Sample as fine as possible; generally at least with a 2 min sampling for photometry,
# n_int=5 was found to be a good number of interpolation points for any short photometric cadence t_exp;,
# increase to at least n_int=10 for 30 min phot. cadence,
# the impact on RV is not as drastic and generally n_int=5 is fine enough,
###############################################################################,
t_exp_n_int_Leonardo,
###############################################################################,
# Number of spots per object and instrument,
###############################################################################,
host_N_spots_Leonardo,
###############################################################################,
# Number of flares (in total),
###############################################################################,
N_flares,
###############################################################################,
# TTVs,
###############################################################################,
fit_ttvs,False
###############################################################################,
# Stellar grid per object and instrument,
###############################################################################,
host_grid_Leonardo,default
###############################################################################,
# Stellar shape per object and instrument,
###############################################################################,
host_shape_Leonardo,sphere
###############################################################################,
# Flux weighted RVs per object and instrument,
# ("Yes" for Rossiter-McLaughlin effect),
###############################################################################,
3) params.csv
The main parameters that need to be modified here from candidate to candidate are the:
b_rr,0.1,1,trunc_normal 0 1 0.1 0.05,$R_b / R_\star$,
b_rsuma,0.2,1,trunc_normal 0 1 0.2 1.5,$(R_\star + R_b) / a_b$,
b_cosi,0.0,1,uniform 0.0 0.03,$\cos{i_b}$,
b_epoch,1.09,1,trunc_normal -1000000000000.0 1000000000000.0 1.09 0.05,$T_{0;b}$,$\mathrm{BJD}$
b_period,3.41,1,trunc_normal -1000000000000.0 1000000000000.0 3.41 0.05,$P_b$,$\mathrm{d}$
b_f_c,0.0,0,trunc_normal -1 1 0.0 0.0,$\sqrt{e_b} \cos{\omega_b}$,
b_f_s,0.0,0,trunc_normal -1 1 0.0 0.0,$\sqrt{e_b} \sin{\omega_b}$,
The full file looks like:
#name,value,fit,bounds,label,unit
#companion b astrophysical params,,,,,
b_rr,0.1,1,trunc_normal 0 1 0.1 0.05,$R_b / R_\star$,
b_rsuma,0.2,1,trunc_normal 0 1 0.2 1.5,$(R_\star + R_b) / a_b$,
b_cosi,0.0,1,uniform 0.0 0.03,$\cos{i_b}$,
b_epoch,1.09,1,trunc_normal -1000000000000.0 1000000000000.0 1.09 0.05,$T_{0;b}$,$\mathrm{BJD}$
b_period,3.41,1,trunc_normal -1000000000000.0 1000000000000.0 3.41 0.05,$P_b$,$\mathrm{d}$
b_f_c,0.0,0,trunc_normal -1 1 0.0 0.0,$\sqrt{e_b} \cos{\omega_b}$,
b_f_s,0.0,0,trunc_normal -1 1 0.0 0.0,$\sqrt{e_b} \sin{\omega_b}$,
#dilution per instrument,,,,,
dil_Leonardo,0.0,0,trunc_normal 0 1 0.0 0.0,$D_\mathrm{0; Leonardo}$,
#limb darkening coefficients per instrument,,,,,
host_ldc_q1_Leonardo,0.5,1,uniform 0.0 1.0,$q_{1; \mathrm{Leonardo}}$,
host_ldc_q2_Leonardo,0.5,1,uniform 0.0 1.0,$q_{2; \mathrm{Leonardo}}$,
#surface brightness per instrument and companion,,,,,
b_sbratio_Leonardo,0.0,0,trunc_normal 0 1 0.0 0.0,$J_{b; \mathrm{Leonardo}}$,
#albedo per instrument and companion,,,,,
host_geom_albedo_Leonardo,0.0,0,trunc_normal 0 1 0.0 0.0,$A_{\mathrm{geom}; host; \mathrm{Leonardo}}$,
b_geom_albedo_Leonardo,0.0,0,trunc_normal 0 1 0.0 0.0,$A_{\mathrm{geom}; b; \mathrm{Leonardo}}$,
#gravity darkening per instrument and companion,,,,,
host_gdc_Leonardo,0.0,0,trunc_normal 0 1 0.0 0.0,$Grav. dark._{b; \mathrm{Leonardo}}$,
#spots per instrument and companion,,,,,
#errors per instrument,
log_err_flux_Leonardo,-7.0,1,uniform -15.0 0.0,$\log{\sigma_\mathrm{Leonardo}}$,$\log{ \mathrm{rel. flux.} }$
#baseline per instrument,
baseline_gp_matern32_lnsigma_flux_Leonardo,0.0,1,uniform -15.0 15.0,$\mathrm{gp: \ln{\sigma} (Leonardo)}$,
baseline_gp_matern32_lnrho_flux_Leonardo,0.0,1,uniform -15.0 15.0,$\mathrm{gp: \ln{\rho} (Leonardo)}$,
We need a function to be provided with the mainly results from TLS and generates the three files to run the allesfitter fit.
To lunch the fit we just need to run run.py
, which looks like:
import allesfitter
allesfitter.show_initial_guess('allesfit')
allesfitter.ns_fit('allesfit')
allesfitter.ns_output('allesfit')
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.