specdb/igmspec

Spectral Database Tools for Probing the Intergalactic Medium

igmspec

Welcome to the igmspec repository. The igmspec database is intended to provide all publicly available spectra that probe the intergalactic medium (IGM). Parties interested in accessing and interfacing with the database should see the primary publication: http://coming_to_arxiv.soon Further documentation is available at readthedocs: http://specdb.readthedocs.io/en/latest/igmspec.html

This repository has two main purposes:

1. Provide additional descriptions of the datasets ingested into the igmspec database.

2. Maintain the software required to build the database.

It is rather unlikely that you will want or need to clone this repository.

igmspec's Issues

Speed things up by breaking BOSS into multiple groups

Not entirely necessary, but would be nice..

Other Surveys

Please add to this list:

• HST STIS/GHRS from Cooksey+10
• HST COS-Halos, COS-Dwarfs
• Gabor's HeII survey
• MagE LLS survey (Fumagalli+ 15); overlaps with HD-LLS
• H100 (Neeleman+13)
• ESI DLAs (Rafelski)
• GRBs galore
• QPQ (eventually)
• HIRES dataset from UCSD (Pro+07)

Notebooks in docs

Slurp the iPython notebooks into .rst files

Have get_igspec script quote the size of the HDF5 file being downloaded

As titled

Ingest HSLA

As titled

Properly ingest Cooksey continua

e.g. 3C273

Version 2

This thread should capture the wish list for v02.

v02 incorporate new datasets, continua and software
to generate proprietary datasets.

v02 will also include a paper to be submitted (or at least posted).

Deal with MUSoDLA MagE

There is a mix of fluxed and normalized data..

Observation Date

Insist on OBS-DATE in v02.

Global co-add

Consider creating a 'global' co-adding code

Include continua for COS-Halos

@jessicawerk will generate them.

Splicing

Add a splice option for multiple spectra

Defs in Meta

Add survey definitions to meta in hdf file

Move repository to specdb Organization

As titled. This will happen when v02 is rolled out.

Headers

Consider including the FITS headers for all non-SDSS/BOSS data.

And even those too..

Version history

Should the code be written to:
a. Enable the "exact" reproduction of every version release of the DB?
b. Build on pervious versions or start over building from scratch?

Cross-check quasars

Against the Meyers list

Will be done in v02

XQ-100 records have wrong IGM_ID

@profxj

In the v02 database, XQ-100 quasars are given wrong (and duplicated) IDs. They span row 382443 to row 382539, but their IDs are from 1 to 97.

And hence after the XQ-100 quasars, the row number of a record is not the same as its IGM_ID. For example row 382540 has IGM_ID 382443.

Would like to get an amended copy of the database even before v03 release.

Name game

Let's decide on the name for this repository/package

JMO: sigma = Spectra of the IGM for Astrophysics

Add h5py to install notes

As titled

Redshift errors in BOSS catalog

e.g. IGM_ID = 302190

Not sure if it is the catalog fault or my own.

Insist zem, RA, DEC, etc in group meta match catalog

Allow for additional coords, zem, etc. in meta table
but insist that zem = zem

Other required meta data

• Grating name

Ingest the Schneider DR7 QSO catalog

Current ingestion is JXP homebrew

Fix GGG DATE-OBS

SMJD is for SDSS

For GMOS either refer to the FITS header of the spectrum or the paper.
Several targets got reobserved, so the FITS header is not always very informative.

z BOSS

Should we use Z_PCA or Z_PIPE or ??

SDSS (Hewett+Wild)

Consider ingesting their (better) sky-subtracted spectra.

Deal with bad XQ-100 coordiantes

Yup

Survey_ID

Make these unique identifiers for the quasars, not the spectra.
GGG, at the least, needs fixing

Multi-slit data?

Suppose, as eventually in the case of HS1700 for example, we have a set of multi-slit data from MOSFIRE, LRIS, DEIMOS, etc near a quasar from our surveys. Should we incorporate those spectra into igmspec? It could facilitate fast searches for galaxies near certain redshifts in the qso spectrum. May be too much work for too specific a return.

Freeze IGM_ID in v01

As titled, after fussing about duplications.

PSFFlux errors in quasar catalog

Likely JFH's error

v03

Add at least the following:

• BOSS DR14
• KODIAQ DR2
• Eilers' QSOs
• First set of GRB spectra

BOSS QSOs with bad coords

e.g. 213248 BOSS_DR12 214.1062 52.5452 2.563 spec-7030-56448-0359.fits.gz

Fix wrong WV_MIN values in ESI data of HD-LLS-DR1

As titled.

Values are zero and should be just below 4000A.

"proprietary" version

Any interest in making a proprietary version of the DB so that we can include "in-work" data (e.g. WFC3 pairs, operation headshot, etc) spectra. perhaps a call to get_igmspec that requires a password (not sure how one would do that....)

BOSS z>4.8

Fill in the (missing) z>4.8 QSOs.

Installation problem

When trying to reinstall specdb from scratch (using python setup.py install), I now get the following error:
(astroconda) ntejos@arrakis:~/python/specdb$ python setup.py install
running install
running bdist_egg
running egg_info
writing specdb.egg-info/PKG-INFO
writing top-level names to specdb.egg-info/top_level.txt
writing dependency_links to specdb.egg-info/dependency_links.txt
reading manifest file 'specdb.egg-info/SOURCES.txt'
writing manifest file 'specdb.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying specdb.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying specdb.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying specdb.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying specdb.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating 'dist/specdb-0.1.dev0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
Traceback (most recent call last):
File "setup.py", line 86, in
setup(**setup_keywords)
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/command/install.py", line 67, in run
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/command/install.py", line 109, in do_egg_install
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/command/bdist_egg.py", line 223, in run
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/command/bdist_egg.py", line 467, in make_zipfile
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/command/bdist_egg.py", line 460, in visit
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/zipfile.py", line 1151, in write
self._writecheck(zinfo)
File "/home/ntejos/anaconda/envs/astroconda/lib/python2.7/zipfile.py", line 1117, in _writecheck
" would require ZIP64 extensions")
zipfile.LargeZipFile: Filesize would require ZIP64 extensions

Not sure if it's caused by specdb or something else...

Update dependences

linetools, etc.

Ingest MUSoDLA in v02

Kindly provided by Murphy & Jorgensen

Refactor to use specdb

As titled. Changes are a coming..

Simple Usage example

Hi all,
I think there is a missing line in the Simple_Usage.ipynb:

I added this to the initial inputs:
from astropy import units as u

Also, when I grab the example spectra there is "sig" keyword missing.
I guess the sigma spectrum is missing?

I ran the cell:

Grab spectra

J0812spec, meta = igmsp.grab_spec('KODIAQ_DR1', ids0812)

and got this in return:
Staged 1 spectra totalling 0.0032 Gb

Loaded spectra

TypeError Traceback (most recent call last)
in ()
1 # Grab spectra
----> 2 J0812spec, meta = igmsp.grab_spec('KODIAQ_DR1', ids0812)

/anaconda/anaconda/envs/astroconda/lib/python2.7/site-packages/igmspec-0.0.dev0-py2.7.egg/igmspec/interface_db.pyc in grab_spec(self, survey, IGM_IDs, verbose, **kwargs)
195 return
196 # Generate XSpectrum1D
--> 197 spec = XSpectrum1D(data['wave'], data['flux'], sig=data['sig'], masking='edges')
198 # Return
199 return spec, self.meta

TypeError: init() got an unexpected keyword argument 'sig'

Nino

Complete Version 1

This Issue announces the release of v1.0.

Cannot build the database using get_igmspec

The error is

(astroconda)ntejos@arrakis:~/python/igmspec/DB$ get_igmspec 
Traceback (most recent call last):
  File "/home/ntejos/anaconda/envs/astroconda/bin/get_igmspec", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/ntejos/python/igmspec/bin/get_igmspec", line 12, in <module>
    import igmspec.scripts.get_igmspec as get_igmspec
ImportError: No module named get_igmspec

Seems some sort of compatibility issue w/r to the old igmspec implementation.

Test sdss_igmspec

Need to add an SDSS spectrum to the debug.hdf5 file

Continuua

Consider adding continua, e.g.
SDSS-DR7 (mine or Gunagtung's)
BOSS (QPQ)
GGG (from LLS analysis?)

Eliminate zeros in continua

Set to 1 instead

Ingest continua of SDSS DR5 and Zhu+14

Hi X, please ingest existing continua of SDSS and BOSS quasars.

wget continue

Modify get_igmspec to enable continue mode

Clean vstack

Add a check before writing that all meta tables
can be stacked using specdb.utils.clean_vstack