astrocatalogs / astrocats Goto Github PK

View Code? Open in Web Editor NEW

40.0 9.0 12.0 134.74 MB

Astrocats package for constructing astronomical catalogs

Home Page: https://astrocats.space

License: MIT License

Python 100.00%

catalog open-astronomy-catalogs

astrocats's Introduction

Astrocats: Open Astronomy Catalogs

The Astrocats package enables astronomers to construct their own curated catalogs of astronomical data with the intention of producing shareable catalogs of that data in human-readable formats. Astrocats is used by several existing open astronomy catalogs, including:

The process for creating one's own open astronomy catalog involves checking out this package and designing a "module" for it that is specific to that catalog's needs, a Wiki is available with instructions for doing so. At the moment the most developed module is the Open Supernova Catalog module; to set up astrocats with the supernovae module, one needs to check out two repositories:

git clone [email protected]:astrocatalogs/astrocats.git
cd astrocats
git clone [email protected]:astrocatalogs/supernovae.git

Astrocats will soon be listed on PyPi, at which point the install instructions will involve pip and a setup script, for now the install must be performed manually.

astrocats's People

Contributors

Stargazers

Watchers

Forkers

jparrent wmwv finzellt crawfordsm mnicholl lib-j zworkb maplehu spamfour alex-w davecoulter astroskip

astrocats's Issues

Improving and expanding the Photometry class

Currently the Photometry class contains only simple primitives, with different types of flux/brightness measures enumerated. This is something that could be both streamlined and expanded significantly. This issue is a place to record ideas and brainstorm.

Streamlining: for example, by using Quantity classes which include errors and units internally
Expanding:
- Using a new, more general 'brightness'/'intensity' class which can readily convert between different flux/magnitude units.
- Allowing conversion between flux and luminosity using the distance
- convert from bands to bolometric and visa-versa using more detailed modeling.

Missing secondary source for McCully 2014 on SN2008A

No secondary source is shown for McCully 2014 for SN2008A, but it is likely to be UCB. Need to figure out why the secondary source is not being added.

Remove the need for the `repos.json` input file

Perhaps an individual catalog doesn't need to use the output-repository structure that supernovae does. For example, the catalog may not version-track the output files at all, they may all be contained in a single repository, or perhaps they want to keep the output in the same repository as the code itself.

Instead of requiring the repos.json file and associated methods, we can just require the necessary API methods. i.e. instead of get_repo_output_file_list() (which is used by the delete_old_entry_files() method --- and associated task), require a more general get_output_file_list() which can store/construct those files in any (reasonable) way they wish.

[SuSpect] Add FITS header files to repository

Original poster, guillochon: I'd like to add the metadata from the SUSPECT spectra to the catalog, and I think the easiest way to do this is to just add all the FITS header files to the SUSPECT folder. That way I can parse the accompanying header file when I read in the spectra.

*migrated issue

Can't use --depth command anymore?

The --depth arg was used to restrict the depth of the repositories checked out, but it appears that it can no longer be passed, unless the syntax has changed to something weird. I tried several combinations and none worked:

python -m astrocats --depth 1 supernovae import

python -m astrocats supernovae import --depth 1

The --depth arg is needed to get the Travis build working, otherwise we run into Travis' disk quota.

+00 declination coordinates incorrectly imported as negative.

This issue affects potentially up to 1,000 supernovae in the catalog, and its cause is known and will be fixed in an upcoming commit. For the moment, do not trust SN coordinates with ±00:xx:xx declinations!

Add recursive type checks for `Key` values which themselves are `CatDict` instances.

Currently, if the value for a key in an Entry or CatDict is itself a CatDict (instead of a primitive; see for example), then a string literal is used as the key instead of a Key instance. This is because the Key class doesn't know how to handle a value which is not one of the (current) built-ins {STRING, NUMERIC, BOOLEAN...}.

Add a way to use Key instances to handle composite class values. Probably the best thing to do is to have the corresponding KeyCollection do the type checking...

Interrupt stalled import tasks

Individual tasks should exit it they are either taking far longer than normal to complete, or are stalled somewhere. In the long-run, it might be good to even set these up as non-blocking subprocesses (or other threading) --- presumably url-connection cycles will be hogging lots of time where the processor/drive is still available.

Distinguish host redshift from supernova redshift

Currently, a single redshift tag is used to mark a SN's redshift, this redshift might be the redshift of the supernova, or it might be the redshift of the host galaxy, depending on the source. A new tag should be created, "hostredshift" that is specifically for this information. When a host redshift is known for a source but the supernova redshift is not known, the supernova redshift should be set to the host redshift, but the supernova redshift should be marked as "derived."

[LSQ12dlf] duplicate photometry points

The V and r photometry seem to be the same datapoints at an offset. In fact there seem to be 2 copies of the V band data, and of many other bands B, I... and the r data looks like it is a copy of the V data at an offset.

Changing bin size on individual pages slightly shifts spectra rightwards

To extinct or not to extinct?

@jparrent Looking to digitize this paper, and they provide both S-corrected photometry and the raw photometry. Do you think it's best to import the corrected or uncorrected photometry into the catalog? (or both?)

duplicated UCB spectrum for 98S

@guillochon I believe there's a bug that is making two spectra for 98S much narrower than they should be. It's the first two spectra at the top from UCB. In the display window, select the pan button, and drag the cursor to see the original greyed out spectra near 6560 Angstroms. The third spectrum down is what they should look like, although the third spectrum down is from Superfit where the presented frame is off, which is what originally caught the eye.

Adding Extragalactic SNRs

I've perused some small databases of extragalactic SNRs and have added several hundred to the OSC, but there are ~2,000 according to SIMBAD. Adding these will be tricky given the names of these SNRs vary wildly and do not conform to a standard naming scheme. We might need some SNR experts to aid us in adding these given the state of the available data.

[SuSpect] Redshift corrections

This is a running list of SN that need to be reset to a proper observer frame. I will do these and submit a PR in a few weeks, but I'm listing them here until they are fully fixed:

Disclaimers:
--it's not all objects from SuSpect, much less all spectra for each object listed
--some of the singles to check may be false alarms, but there are no other sources by which to compare (e.g., CfA).

Convert SNDB import to be from API rather than scraping

Isaac Shivvers has nicely written an API for us to pull photometry from SNDB, currently we still use the old scraping algorithm. We should switch to the new API ASAP.

errors in the brightest absolute mags

GRB 080319B: M = -39 -- this is the brightness of the GRB. The associated SN peaked at around -19 (http://iopscience.iop.org/article/10.1088/0004-637X/691/1/723/meta, http://iopscience.iop.org/article/10.1088/0004-637X/725/1/625/meta)

SN 2013cq: M = -29 -- also a mix of SN and GRB magnitudes

SN 2013ez: M= -29 -- also a GRB. I see a pattern emerging!

SN 2013fu: M= -28 -- GRB

"Possible supernova": M= -28 -- Redshift highly uncertain, and may be lensed

SN 2008hw: M= -26 -- GRB

SN 2004cq: M= -26 -- two conflicting redshifts, the one being used is incorrect

SN 1998ev: M= -25 -- apparent mag wrong, should be 21.3 not 14

CSS140424:123226+154840: M= -25 -- the brightest point is unrelated to the SN; it is an artefact at the edge of the CSS detector. This point is about 2 years after the actual SN. Note also that the for the 'claimed type', there was an ATel on this that would be a much better reference than CRTS or Latest Supernova pages. I don't know if this is a general trend, but ATels give a lot more information than these other sources. If possible, I would strongly encourage checking ATels for every SN and linking to the catalogue page!

SN 2010ma: M= -24 -- GRB

LSQ12axx: M= -24 -- Redshift on Latest Supernova page (0.399) is wrong, should be 0.0399 from ATel

SN 2014W: M= -24 -- Redshift on Latest Supernova page is wrong (can't work out where the number they have came from). CBET gives recession velocity rather than redshift, which I guess complicates reading things in directly from there. Corresponds to z=0.036

2001iw: M= -24 -- stumped by this one. The redshift is correct. Multiple SN cosmology papers quote the observed peak as 22 mag (you've imported this point). This looks correct for a Ia. However, the R and I band light curves peak at around 17 mag. The catalogue quotes the sources as 1,2. Source 2 is the actual paper (Barris et al), and the light curves look like they peak at 22 mag. I can't see any tabulated data. However in source 1 (Sternberg catalogue) where it seems your photometry has come from, the light curves have been shifted up by about 5 mag! I hope this isn't a common problem with light curves from this source... Or that not many of your other light curves come from there!

2002ab: M= -24 -- oh dear, looks like exactly the same issue.

2001bu: M= -24 -- took a bit of work to figure this one out! Redshift is from 'Unified SN Catalogue' (Lennarz et al). The host designation has a Q in it, which it turns out means quasar. So I think this must be a quasar rather than a SN. Might be worth going through everything from that catalogue and looking out for untyped SNe with a Q in the host name!

2001ca: M= -24 -- also from Lennarz catalogue. Not listed as a quasar, but I also see that this catalogue flags things that were never confirmed as SNe. Again, may want to filter out objects like this: one detection, never confirmed as SN, redshift also flagged as uncertain...

PS1-12qh: M= -24 -- ATel says its a QSO, not SN or LBV. Latest SN page has also got the magnitude wrong; should be 19.2, not 18.6

SN 2001iy: M= -23.7 -- see 2001iw and 2002ab

SCP06F6: M= -23.6 -- nothing wrong with the data or redshift. Maybe this isn't too important, but the absolute mag here is exaggerated simply because we need a fairly hefty K-correction at this redshift. You might want to implement M = m - DM + 2.5log(1+z) to get more realistic absolute mags for high-z SNe.

2001jp -- see 2001iw etc

2011jb -- couple of bad points in CSS light curve throwing peak mag way off

2010hu -- also a weird CSS point unrelated to SN

2001fo -- another for the weird Sternberg subclass of error

That's all I have time for for now, more soon...

Photometry and Spectra Icons

It dawned on me today that the photometry and spectra icons over at sne.space take the user to the same spot. I understand the displays are under construction, but the icons could be merged unless keeping them separate would be better in the long run.

[SN2004dn] <Descriptive issue title>

Sams as the comment on 04dk, and I think this will be true for all SNe with both Drout11 and CCCP photometric references: there are 2 V and 2 R band photometries, that differ by 1 magnitude, but they are both CCCP photometry: the same data, different reduction. One of them must be superseded. I think the Drout11, which is published, should be final, and the CCCP one should be the quick and dirty reduction, but I am not sure, and there is no date on the reduction webpage.

[SuSpect] Trimming via "exclude"

This is a running list of objects that have spectra in need of a shave. I will add "exclusion" regions to the json files after redshift corrections are complete.

[SN2004dk] <Descriptive issue title>

there are 2 V band photometries, that differ by 1 magnitude, but they are both CCCP photometry: the same data, different reduction. One of them must be superseded. I think the Drout11, which is published, should be final, and the CCCP one should be the quick and dirty reduction, but I am not sure, and there is no date on the reduction webpage.

[UCB] Error for one epoch of 90M spectrum

Thank you to Isaac Shivvers and Jeff Silverman for the clarification on this one. They found the same issue some time ago:

https://sne.space/sne/SN1990M/
The top-most spectrum is being posted with an epoch of -75, when it should instead be closer to +216 according to Silverman et al. (2012) -- 2012MNRAS.425.1789S

The affected file is:
sne-external-spectra/UCB/sn1990m-19900401-opt.flm

From the looks of it, it's definitely at day +200-300, and there's no such thing as -75 for a SN Ia. For 87A, yes.

Given that there is a legitimate misprint in the date for this spectrum, and that 1991 would make sense instead of 1990 (private comm, IS, JS), I think it would be fine to change the year to 1991 in the filename. But we can discuss later before I go and change anything. At any rate, hello future SN enthusiasts, watch your step.

Add band used to determine date of maximum to metadata

Implementing setup and installation

This is a place to discuss / brainstorm how the setup script and installation of astrocats and catalogs should work in the future.

One of the first priorities should be making a test-catalog more easily runnable, and clearly show important/helpful results: see e.g. #74 (comment)

[SN 1983G] Wrong reference

The spectra of 83G from SuSpect do not have the correct reference because SuSpect does not have the correct reference. Instead the pointer is to Cristiani et al. (1992), which is on SN 1986G, not 83G. Searching for the original work brings up nothing, and the last hint from SuSpect is that the data were donated by the Asiago SN Group. http://graspa.oapd.inaf.it/

Fortunately I was able to use SNID to determine that the wavelength solution is fine and in the observer frame for the files.

BTW there really is a SN 1983G, but the spectra can only be found through GELATO (another SNID-like tool): https://gelato.tng.iac.es/. Which means the spectra can only be seen after submitting a spectrum for comparison.

https://gelato.tng.iac.es/templates/
"The templates database is continuously growing with new additions, which are immediately available to all GELATO users. GELATO currently uses 2926 template spectra to classify your SNe."

"Available" in GELATO, not available for download.

Subcommands don't show 'help' dialog from argparse

Need to find a way to get the subcommand parsers to show their own help, when -h/--help is passed after the subcommand name.

e.g.:

astrocats -h should run the top-level help.
astrocats -m catalog -h should run catalog-level help.
astrocats -m catalog import -h should run the import help.

Convert import and cleanup tasks to their own classes; separate behavior from entry and catalog

We have a Task object defined in astrocats/catalog/task.py which represents a to-do-task. These to-do-tasks include a function name (and the submodule they're located in) which should be executed to complete the given task. Instead, the Task object should be expanded to include the function itself. Likely, this should be something like:

class Task:
    def __init__(self, ...):
        # setup all of the variables currently in the `Task` objects

    def load(self, ...):
        # The actual function to complete the task

A list of Task objects will be created, and each will just have a few parameters (like they do now). If a Task is active (i.e. Task.active == True) then Task.load() will be called in the import script.

Benefits:

Make it easy to subclass the general-catalog tasks as needed by individual catalogs.
Remove the need for a tasks.json input file. Instead all of the tasks can live in the tasks directory, which will be searched. The default settings will be stored in the class definitions.

toggle switch for black spectra

The colored lines are helpful as is, particularly for catching misaligned spectra, but could there be a toggle switch in the spectrum figures for the line colors to all turn black? I suggest this now, for later, but for inspecting the evolution of certain features, it would really only be visually appealing in black after the data have been cleaned up a bit. Getting there.

[SN 2003aa] Ic, not Ia

@guillochon So it turns out this one was being flagged as a Ia, when it's a Ic.

SUPERNOVA 2003aa IN NGC 3367
Further to IAUC 8063, T. Matheson, P. Challis, and R. Kirshner
report that a spectrum (range 370-750 nm) of SN 2003aa (cf. IAUC
8063), obtained by P. Berlind with the 1.5-m reflector on Feb. 2.31
UT, shows it to be a type-Ic supernova. The spectrum is similar to
that of SN 1994I, four days before maximum (Filippenko et al. 1995,
Ap.J. 450, L11).

I flagged the claimedtype of Ia as erroneous and wanted to update the value to Ic. I thought I could do this via the pencil icon but then got an error. What did I miss?

Add bibcodes for CfA spectra

Currently the only source referenced for spectra that appear on the CfA's page is the URL to the page itself, however we do have the publication info available from that page. This should be added for completeness.

supernovae from the future

Sorting by discovery date (most recent first), shows a few SN with incorrect discovery dates in the future:
SN 2010kb, SN 2010ka, SN 2010jz, SN 2010jy, SN 2010jx, SN 25747NSV

[SN2013df] missing data and reference

there is VanDyk+ 2013 reference (3) but i do not see the NIR photometry, or any optical photometry associated to reference (3) a significant amount of photometry available for this object. And there is a missing reference to Morales-Garoffolo+ 2014 (some of their data is also in Brown 15, but not all). I will try and upload it.

Add http://arxiv.org/abs/1511.06778

Radio data (mostly) black in color

Currently there is no color table assigned to radio data, which is input as frequencies rather than named bands, so most radio data currently appears as black regardless of frequency. We will add a continuous color table to the radio data shortly.

[2011fe] date of maximum

@guillochon Its max B-band date can be updated from 2011 Sep 6 to 2011 Sep 10 (MJD = 55814.51).

Pereira et al. 2013
http://adsabs.harvard.edu/abs/2013A%26A...554A..27P

Add PESSTO Spectra/Metadata

SN2013dy currently missing from catalog

This is the one file in excess of 100mb uncompressed, seems it is getting deleted by one of the auto-compression scripts. Hopefully will return in next daily update but keeping this issue open until it is resolved.

[SuSpect, UCB] Spectra have unknown flux unit

Spectra will need to be scaled to the photometry.

[Additional Spectra] R. Foley repo

https://github.com/astrofoley/astr596/tree/master/data/spectra

This is a pile of spectra that needs to be sifted through. I will do this and will supply what the catalog does not already have.

Naming errors

Received an e-mail from Victor Lebedev highlighting some naming errors, which we need to address:

I found a few bugs and typos in your supernova catalog:

two supernovae ( SMTJ21413915-5643445, MASTEROTJ134201.21-023856.2 )
present twice;

The following supernovae in alias mentioned twice:
Gaia16agf
MASTEROTJ111920.95+724857.3
PSNJ18121900+2131150
SNhunt10
SNhunt138
SNhunt27
SNhunt28
SNHunt309
SNhunt313
SNhunt314
SNhunt33
SNhunt34
SNhunt36
SNhunt39
SNhunt41
SNhunt42
SNhunt45
SNhunt49
SNhunt51
SNhunt55
SNhunt58
SNhunt62
SNhunt63
SNhunt64
SNhunt66
SNhunt7
SNhunt70
SNhunt77
SNhunt79
SNhunt82
SNhunt84
SNhunt85

supernovae SNhunt133, SNhunt281
are doubles PSNJ17124620+2313265, SN2015bp

in a supernova AT2016aha erroneously alias name SNhunt307

supernova PESSTOESO154-G10 one hand you have a double SN2014eg , while in the NED is designated as SN2013fc

Finally, "Possible supernova" there can be named supernovae (and Name , and Alias).

[SuSpect SN types] Check with SNID

Below is a running list of objects with classifications and/or epochs to double check with SNID.

92ac -- Date of maximum appears to be off by ~20 days. This would place peak brightness at May 9 1992 instead of May 29.

`add_*` functions in `test` should use `KeyCollection` members rather than literals

So currently when we do something like say add_photometry, we pass a list of kwargs to the function where each kwarg is really a string:

catalog.entries[name].add_photometry(time=mjd, band=band, magnitude=magnitude, 
    e_magnitude=e_magnitude, source=source)

But, since say magnitude could be changed to a different string later, it might be better to pass the kwargs as a dictionary, with the dictionary members pulling from the KeyCollection members, like so:

catalog.entries[name].add_photometry(**{PHOTOMETRY.TIME: mjd, PHOTOMETRY.BAND: band, 
    PHOTOMETRY.MAGNITUDE: magnitude, PHOTOMETRY.E_MAGNITUDE: e_magnitude, 
    PHOTOMETRY.SOURCE: source})

This makes the add_* calls a bit longer but also more stable to future changes. Should we adopt this as a best practice?

SNDB supernova types not parsed properly

Currently the front-end shows non-standard supernova types when they are pulled from SNDB (e.g. Ia-norm, etc.), this has been fixed in the code but won't be reflected in the catalog for a few more hours until the import script finishes. I will close this issue once the script completes.

SN1987A photometry mismatch between sources

@jparrent I'd like to figure out exactly why the SUSPECT and the data pulled from Sternberg seem to disagree in regards to the photometry, which is most prominently visible in the I-band (almost a half mag difference between the two sources), but is also apparent in the other bands as well. As far as I can tell, both Sternberg and SUSPECT pulled this data from the same batch of papers from Whitelock and Catchpole. Any idea why the two are different?

[SN 2003bg] lone spectrum misaligned

I'm wondering if issue #32 is a bug.

sne-external-spectra/Suspect/II_spec_ascii_20110715/2003bg/2003bg_20031129_3800_9324_00.dat

Similarly for 03bg there is a lone spectrum that is misaligned from where it should be. In the plot on the 03bg page, it's the third from the bottom that's too far shifted toward the red. All the files are in the correct observer frame, yet one of them is off--the one that is off has the correct numbers in its file.

Type IIP and II-Pec both being converted to IIP type, when they are distinct

This is also an issue on the about page. Have fixed in the code and will close the issue when the fix is live.

Add epoch relative to rest-frame maximum to spectra tooltips

[Catalog Table] Sorting by luminosity distance

Just noticed this. When I order SN Ia ('Ia') in the table at sne.space by luminosity distance (nearest events at the top, so increasing distance downward), the events that do not have a distance are returned first. Is there a way to have the events with NULL or empty entries go somewhere else when sorted? I imagine it would have to work both ways too, either increasing or decreasing luminosity distance. For example, page 12 is when the first nearest distance measurement shows up (SN 1604A). The page flipping is fast enough that it's still readily usable, but just a heads up.

[SN 1990I as in 'eye'] lone misaligned spectrum

sne-external-spectra/Suspect/Ibc_spec_ascii_20110715/1990I/1990I_19900608_2979_10596_00.dat

For some reason this file for SN 1990I is misaligned, and I don't think it should be. I came across this one going through my list and when I plot the same file it is not misaligned wrt to the other files; i.e. the file is in the [inappropriate] rest frame, but it is out of sync against the other spectra which are [given] in the rest frame as well. Because this issue is a mouthful of correct/rest/observer frame, we can revisit this together after I am done correcting the others, but just thought I'd mention it here.

[Superfit] Redshift corrections, dupes, and epochs

I believe I'm coming to a consensus on the Superfit spectra. It's fine if and when they are shown, but:
--they appear to be rest frame spectra, so they need not be corrected for redshift in the import script. There may be exceptions, but for the most part this is true, and I'm tracking the exceptions.
--the epochs for some spectra appear to be the negative of what they should be, e.g., day -116 for 90aa I'm guessing should go to +116. This seems to be the case for others, and the underlying thing here is that there are a few naming conventions for the superfit files. Anything with a "m05" in the filename refers to day -5 in the usual sense, but for post-max, there are two different conventions, "u" and "p", both apparently meaning +number of days since maximum light.