afeinstein20/eleanor

A tool for light curve extraction from the TESS FFIs.

Hi there 👋

I'm Adina. I'm a NASA Sagan Postdoctoral Fellow at the University of Colorado Boulder trying to understand the interaction between baby stars and planets!

🔭 I’m currently working on

• Characterizing stellar flares in the ultraviolet and understanding their impacts on close-in exoplanets
• Developing open-source software to reduce data from the James Webb Space Telescope!

💭 I'm currently thinking about

• Using stellar flare rates as a new metric to constrain stellar ages
• Disentangling chemical signatures of a star from planet

📫 How to reach me:

• adina.feinstein[@]lasp.colorado.edu

eleanor's Issues

Clean-up the ellie directory

There shouldn't be so many generated files in the ellie directory. The text files, movies, and figures should be moved to directories like paper or data or removed entirely.

Travis failing

@nksaunders can I task you with updating the tests so it passes? I believe it should at this point, everything (except PCA for now, but there's no test for that?) should work for sector 1. I removed the multi-sector test, and I think the others should work but we're getting a failure right now!

Documentation does not include install instructions

The eleanor docs page here: http://adina.feinste.in/eleanor/ does not seem to include instructions on installing eleanor.

Passing RA/Dec pair

Right now this has to be given in degrees. It would be useful to be able to pass through an astropy skycoord object, so we can be agnostic about format! In the code, we could then turn that number into degrees and do everything else in the code exactly the same way that it's presently done.

Multi-column info in postcards

Here's our stuff from paircoding:

dtype = [
    ("TSTART", np.float64),
    ("TSTOP", np.float64),
    ("BARYCORR", np.float64),
    ("FLUX", np.float32, postcard.shape[1:]),
]
new_postcard = np.empty(len(postcard), dtype=dtype)
new_postcard["TSTART"] = times[0]
new_postcard["TSTOP"] = times[1]
new_postcard["BARYCORR"] = times[2]
new_postcard["FLUX"] = postcard

TIME?!

FFIs all have uniform time and the FOV is so big that BJD varies by a couple of minutes across the detector. We have to update times in the TPFs to account for this.

Cannot install on windows

Installation on windows is broken due to the dependency on fitsio, whose setup.py has multiple problems on windows. That project has had a ticket open about the issue since 2016.

Deleting the fitsio dependency from eleanor's setup.py allows installation to proceed (having cloned from github and running setup.py)

However, running the quickstart example then fails (I have opened a separate issue for that)

FITS headers fixes

In the saved TPF:

• "Sector" is stored twice, once as a string and once as an int. Is there a pressing need to save as one or the other, or can I delete the one of my choosing (the string)?
• "Version" number is hard coded into make_postcards. eleanor.version exists!
• Now that we have flexible background estimation region sizing, we should add that into the header too.

Difference in LC and SC cadences

In sector 1, there are 1282 30-minute FFIs.

In sector 1, there are 20076 2-minute data points in the PDC light curves.

1282 * 30 != 20076 * 2.

Why not? Are we missing 50 FFIs, or is TESS making 846 data points outside of the FFIs that we don't get? I need to figure out when these are happening and then probably ask Roland on twitter.

audit import statements

There are quite a instances of importing packages inside a function. We should audit these and make sure it's really beneficial to do this. I would err on the side of putting all import statements at the top unless it's a function that's really rarely used and a package that takes a while to import. I noticed this issue most in targetdata.py.

Pointing model from other spacecraft data

Following on from a conversation with Adina: The TESS team now publishes their quaternion data. I wonder if we can use that to make a better (more accurate, equally accurate but more stable, faster to produce, or otherwise just alternative) pointing model

Check to see if we already have the targetdata object

Right now, eleanor re-makes the TPF any time it is requested. We do the smart thing with the postcards, to check to see if they are downloaded first before blindly downloading them to save bandwidth. That's good! We could do the same thing with the TPFs, like lightkurve does.

Move postcard background images to seperate files

Since we're using a post processing step to compute the backgrounds, it's probably better to save them as a separate file so we don't risk corrupting the generated postcards. How do y'all feel about that as an option? It would change the user code a bit but perhaps not too much?

save_postcard = False doesn't not save postcards

Since we wrote this, we ported the management of postcard files to astropy. They're now in the astropy cache using their naming convention, so we need to capture them and remove them if save_postcard == False.

Splitting project into two repos?

Hey team, what do you think about also splitting the make_* scripts into a separate repo (that perhaps depends on the main eleanor repo). This would be nice because then not every user would need to compile the new interpolation routine (only if they want to generate the data files) and I think it might be easier to maintain. We could call it eleanor-tools or something... What do y'all think of that? If you agree that that could be a good idea, I'm happy to do it and then transfer ownership of the repo back to you @afeinstein20.

Re-naming

When we re-name the project to elanor, we'll need to do it in a few places and it'll probably break things. First, we'll have to change ellie to elanor in a few places in setup.py. Then we'll need to rename the ellie directory to elanor. I think that it would also be good to rename the git repo, but I'm cool either way with that one.

Best* TPF-level background subtraction

It would be cool if a user could pass through a tuple for how big a region they want to use for tpf(ish)-level background subtraction, which takes that cutout, measures the background, and that's all it does, then tosses that region away and does everything else normally. If this isn't set, then it uses simply their requested size for their TPF to make the background subtraction, as it's done now. I think this would be neat so I'm going to assign myself and think about it.

.load()

Need to add the postcard name to the header so users can re-save files

cannot do .save() more than once due to removal of post_h from original header

Issue with downloading postcard

Kernel keeps crashing when trying to download the postcard (using Jupyter) as can be seen in the screenshot.
When I use the terminal instead, python just quits unexpectedly. Not sure why. I am using Python 3.6.

Detector corners

If someone (me) tries to make a TPF, and the TPF would extend beyond the edge of the detector, there is an error in trying to match up sizes of things.

This worked at one point, so recent changes must have messed up our previous fix somehow.

Example: tic 79015293 in sector 1.

KeyError: "Key 'FFIINDEX' does not exist."

Thanks for putting this package together. It's a huge help.

I'm running into some error which I don't understand. I run the following code in my jupyter notebook:

star = eleanor.Source(tic=2758565, sector=2); data = eleanor.TargetData(star)

and I get the exception in the subject line. I'm using python 3 and eleanor version 0.2.1. I've pasted a screencap of the error below.

Thanks for your help.

Stars not on the detector being found

David noticed that when he gives the coords (152.86113, -38.63042) this does return a position on the CCD and then fails in doing a good job correcting the WCS (which makes sense, halfway across the sky). Why does this one get found in the guide? I don't know but we should figure it out!

Cache directory?

@nksaunders: what's the deal with the .cache/v/cache directory? Apparently you added it, but I'm not sure what it's there for...

What quality flags to take from TPFs?

The FFIs have almost no quality flags set. There are useful flags that should be set we can take from the TPFs, which should be applicable to the entire detector at once (for example: "telescope not in fine point mode." I need to figure out which quality flags are the informative ones for the FFIs. Which means I need to finally understand what argabrightening is.

postcard_guide re-imagining

Right now, the postcard_guide routine finds, if a star is on multiple postcards, the distance from the center of the postcards to the star, and chooses the one where the star is nearest to the center. This is fine when the postcards are on a grid and all the same size, but in the real world we should choose the one where the star is furthest from the edge of a postcard, rather than closest to the center.

Query MAST to see if it has our light curve

Not an issue right now but eventually we might want the code to have the ability to pull a TPF from MAST once they are hosting our data

Problem with tutorial: "FFIINDEX does not exist"

Hi! I have just downloaded eleanor via pip3 and am running iPython with version 3.7.3. When I attempt the data = eleanor.TargetData step of the tutorial, I receive the following errors:

Any thoughts?

Fixing header times

The times in the headers are only consistent with the times for one FFI. It should contain the values the entire time range contained in the file. We use these times in the archive to make it easy for people to find the data so we really need them to be representative of the data in the file.

Quickstart example fails

Running through the quickstart example fails when eleanor.TargetData() is called, throwing an IndexError. Line 197 of postcard.py is trying to access index [4], whereas the postcard fits file that has been downloaded only has entries up to [3] when opened using astropy.io.fits.

My installation was by cloning from github and running setup.py on python 3.7.3 (pip install was not an option on my local machine - see the other issue I have just opened). The offending line appears not to exist in the version that pip installs (which I have managed to do on an inconveniently remote linux server running python 3.6.8)

Targets on the edge apertures

Create new off-centered apertures for targets that are near the edge of the FFI

Visualizing Postcards

Hello again!

The documentation says that eleanor.Visualize "allows for plotting of both postcards and tpfs." However, I have just tried to create a postcard object (the same star from the tutorial) and open it with Visualize, and received this error: 'Postcard' object has no attribute 'tpf'.

Here is a screenshot:

I am trying to gear up to use the custom aperture pixel selection tool, but can't figure out what I'm supposed to load into it to select the pixels!

Thank you!

CADENCENO flags

Ethan Kruse has suggested a table of CADENCENOs to be included, in addition to the timestamps. This is something that will exist in the FFI data starting from sector 5, apparently. I am cool with us putting it in at that point but we should not spend too much time thinking about it now---it will be easier for us to understand their numbering convention and add this in at that future time when we reprocess sectors 1-4 at the end of the year. Putting it here so we do not forget!

data.save

save changes the header and then trying to save a file after you've already saved (but before re-running targetdata) causes an issue. This could be important if you're saving, changing some variables (like pca modes or psf tools) and rerunning. The solution is in self.set_header and Adina knows hwo to fix

Sector not available?

I'm trying to retrieve data for a position covered by Sector 7 so I know the data is available but when I try:

star = eleanor.Source(coords = (ra, dec), sector = 7)

I get:

star = eleanor.Source(coords = coord, sector = sector)

File "/Users/mjg/anaconda3/lib/python3.6/site-packages/eleanor/source.py", line 169, in init
self.locate_on_tess() # sets sector, camera, chip, postcard,
File "/Users/mjg/anaconda3/lib/python3.6/site-packages/eleanor/source.py", line 236, in locate_on_tess
self.locate_on_chip()
File "/Users/mjg/anaconda3/lib/python3.6/site-packages/eleanor/source.py", line 214, in locate_on_chip
raise SearchError("Sorry, this sector isn't available yet. We're working on it!")
eleanor.utils.SearchError: Sorry, this sector isn't available yet. We're working on it!

What is wrong?

Problems with TargetData

Hi all, I am following the tutorial and when I reach the sentence data = eleanor.TargetData(star, height=15, width=15, bkg_size=31, do_psf=True, do_pca=True) the following errors appear. I am not sure what's going on. I have python 3.6.7. Hope you can help me. Thank you.

Repo size is 188 MB

I think there may be a bunch of large deleted files in the git history. We should probably run BFG at some point to remove them.

Check hidden directory for file first

In source.py, we should think about whether we want to check the hidden /.eleanor directory for the target file before going ahead and getting all the data from astroquery, guide, etc.

Error Downloading Target in CVZ using Gaia ID

I'm trying to query a Gaia star. eleanor locates the TIC using eleanor.source, but when I try to access the lightcurve using eleanor.TargetData(star,...), the following error is thrown:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
~/anaconda3/lib/python3.6/site-packages/astropy/io/ascii/core.py in _convert_vals(self, cols)
    952                 try:
--> 953                     converter_func, converter_type = col.converters[0]
    954                     if not issubclass(converter_type, col.type):

IndexError: list index out of range

During handling of the above exception, another exception occurred:

...

~/anaconda3/lib/python3.6/site-packages/astropy/io/ascii/core.py in _convert_vals(self, cols)
    967                     last_err = err
    968                 except IndexError:
--> 969                     raise ValueError('Column {} failed to convert: {}'.format(col.name, last_err))
    970 
    971 

ValueError: Column year failed to convert: invalid literal for int() with base 10: '<h'

I have astropy 3.1.2. Maybe this is due to differing astropy versions?

Issue with saving TPF

When you make a TPF on a different machine and go to re-save using eleanor.TargetData.save(), it looks for the self.post_obj.header to save, which is not on the machine. Create an option that if self.post_obj == None, use header from original TPF

flux_bkg propagated through correctly when using tesscut frames

I think it's being estimated correctly, but somewhere along the way it doesn't get stored in the dataproduct, or gets overwritten by zeroes before it's stored. Writing this down here so I remember to dig into it and track down why later!

Dealing with bright neighbors

Some (many?) of our faint targets will have brighter neighbors. We should make sure they don't get included in our apertures by mistake. This shouldn't happen, because a star partially in an aperture should produce a noisier light curve. But what if the aperture is big enough to include both stars fully?

One idea: If there is a nearby brighter star (possibly read: if the brightest pixel in np.sum(data.tpf, axis=(1,2)) isn't one of the central 9 in the middle of the TPF), restrict our automated aperture generation to only the smaller apertures, things that are up to ~9 pixels in size.

Postcard.wcs not implemented

"pointing_model_per_cadence" doesn't run

When I run pointing_model_per_cadence on ffi, it fails because of a few different bugs (some of which I will fix in an upcoming pull request) but others that are deeper than I'm willing to go. The place where it's stuck now is when it calls tic_by_contamination.

repeated code

There are a few functions that seem to do almost but not quite the same things as each other. We should fix this and make sure there's just one central function to do each task for the sake of clarity.

Specific cases I see are:

• use_pointing_model defined twice in ffi.py
• apply_pointing_model defined in ffi.py and in targetdata.py
• crossmatch_multi_to_gaia, crossmatch_multi_to_tic, find_by_position in mast.py contain duplicated code and return slightly different tables; would be best to standardize table contents and column names (maybe with an add_source_to_table function?)

Can't install on Python 3.7 due to tensorflow dependency

Don't know if this is a known issue or not, but I couldn't find anything in your docs or mentions here: you're currently incompatible with Python 3.7 because tensorflow doesn't have a 3.7 package yet. I tried installing via pip and it failed when it tried to install tensorflow as a dependency. (See here.)

It's not your fault, it's pretty absurd that 3.7 has been out for 6 months and they don't have a package running on the latest Python version (that things like anaconda and miniconda are installing as the default), but here we are.

So I guess on your end, either explicitly put Python 3.0 <= version < 3.7 as your python version requirements (or whatever the min version you support is) and make some sort of note on your install page and/or elsewhere in the docs.

Make guide for multiple camera/chip combinations

Running make_guide on one camera/chip works, but running on a full sector does not because there are some keys that depend on the camera or chip (for example 3-2 has the key TMOFST32 and 4-1 has the key TMOFST41). What do y'all want to do about that?

undefined download directory in postcard.py

While following the tutorial, running
data = eleanor.TargetData(star, height=15, width=15, bkg_size=31, do_psf=True)
throws up an error in line 63:

eleanor/eleanor/postcard.py

Line 63 in 85dbc8a

'working directory instead.'.format(download_dir))

saying download_dir is not defined.

Replacing download_dir with self.post_dir temporarily solved the problem.

eleanor.Source() Not working

I am trying to find a star using eleanor.Source(coords = (), sector = , tc = True), and Python is returning an attribute error:
'AttributeError: module 'eleanor' has no attribute 'Source' '
I updated the eleanor module, so my version should be current. Not sure what the problem is?

Cannot query target with eleanor.Source

None of the three methods in the tutorial for querying a star works. I get the following errors:

>>> star = eleanor.Source(tic=38846515, sector=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/anders/Uni/software/eleanor/eleanor/source.py", line 161, in __init__
    self.coords, self.tess_mag, self.tic_version = coords_from_tic(self.tic)
  File "/home/anders/Uni/software/eleanor/eleanor/mast.py", line 151, in coords_from_tic
    ticData = Catalogs.query_object('tic'+str(tic), radius=.0001, catalog="TIC")
  File "/home/anders/miniconda3/lib/python3.7/site-packages/astroquery/utils/class_or_instance.py", line 25, in f
    return self.fn(obj, *args, **kwds)
  File "/home/anders/miniconda3/lib/python3.7/site-packages/astroquery/utils/process_asyncs.py", line 26, in newmethod
    response = getattr(self, async_method_name)(*args, **kwargs)
  File "/home/anders/miniconda3/lib/python3.7/site-packages/astroquery/utils/class_or_instance.py", line 25, in f
    return self.fn(obj, *args, **kwds)
  File "/home/anders/miniconda3/lib/python3.7/site-packages/astroquery/mast/core.py", line 1922, in query_object_async
    coordinates = self._resolve_object(objectname)
  File "/home/anders/miniconda3/lib/python3.7/site-packages/astroquery/mast/core.py", line 866, in _resolve_object
    raise ResolverError("Could not resolve {} to a sky position.".format(objectname))
astroquery.exceptions.ResolverError: Could not resolve tic38846515 to a sky position.

>>> star = eleanor.Source(coords=(68.959732, -64.02704), sector=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/anders/Uni/software/eleanor/eleanor/source.py", line 153, in __init__
    self.tic, self.tess_mag, sep, self.tic_version = tic_from_coords(self.coords)
  File "/home/anders/Uni/software/eleanor/eleanor/mast.py", line 167, in tic_from_coords
    tess = crossmatch_by_position(coords, 0.01, 'Mast.Tic.Crossmatch')
  File "/home/anders/Uni/software/eleanor/eleanor/mast.py", line 138, in crossmatch_by_position
    return jsonTable(json.loads(outString))
  File "/home/anders/Uni/software/eleanor/eleanor/mast.py", line 77, in jsonTable
    for col,atype in [(x['name'],x['type']) for x in jsonObj['fields']]:
KeyError: 'fields'

>>> star = eleanor.Source(gaia=4675352109658261376, sector=1)
Created TAP+ (v1.0.1) - Connection:
	Host: gea.esac.esa.int
	Use HTTPS: False
	Port: 80
	SSL Port: 443
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/anders/Uni/software/eleanor/eleanor/source.py", line 158, in __init__
    self.tic, self.tess_mag, sep, self.tic_version = tic_from_coords(self.coords)
  File "/home/anders/Uni/software/eleanor/eleanor/mast.py", line 167, in tic_from_coords
    tess = crossmatch_by_position(coords, 0.01, 'Mast.Tic.Crossmatch')
  File "/home/anders/Uni/software/eleanor/eleanor/mast.py", line 138, in crossmatch_by_position
    return jsonTable(json.loads(outString))
  File "/home/anders/Uni/software/eleanor/eleanor/mast.py", line 77, in jsonTable
    for col,atype in [(x['name'],x['type']) for x in jsonObj['fields']]:
KeyError: 'fields'

The object jsonObj is simply a dictionary with the content:
{'status': 'ERROR', 'msg': 'Cannot open database "TESSINPUTCATALOG_V80" requested by the login. The login failed.\r\nLogin failed for user \'webaccess\'.'}

I am using eleanor 0.1.8 with Python 3.7.3. The errors are perhaps related to the new version of TIC, TICv8, that was recently placed on MAST?

Targets that fall in multiple sectors

Simple ways to find when a star falls in multiple sectors (this will be a TESS GI tool possibly?) and stitch these together