Giter Site home page Giter Site logo

mwdust's Introduction

mwdust

Dust in 3D in the Milky Way

image

image

image

image

Contents

Installation

Install the latest released version using pip:

pip install mwdust

To install the latest development version, clone the repository and do

python setup.py install

or

python setup.py install --user

Using custom implementations of necessary HEALPIx functions, basic evaluation of extinction is available on all platforms (Linux, Mac OS, Windows) for all dust maps. However, some HEALPIx-based features like plotting require healpy, which is unavailable on Windows. Install on Linux/Mac OS for full functionality.

Dust Data

By default, dust maps are download when you use them for the first time. If you define an environment variable DUST_DIR, then all dust data downloaded by the code will be downloaded to this directory. If you do not set the DUST_DIR variable, then mwdust will download data to ~/.mwdust.

The code can download all of the necessary data at by running

from mwdust import download_all
download_all()

Note that some of the maps are very large (multiple GB) and some of the downloads are slow, so this may take a while.

The data are put in subdirectories of a directory DUST_DIR or ~/.mwdust, with roughly the following lay-out:

$DUST_DIR/
   combined15/
      dust-map-3d.h5
   combined19/
      combine19.h5
   green15/
      dust-map-3d.h5
   green17/
      bayestar2017.h5
   green19/
      bayestar2019.h5
   maps/
      SFD_dust_4096_ngp.fits
      SFD_dust_4096_sgp.fits
   marshall06/
      ReadMe
       table1.dat
   sale14/
      Amap.dat
      ReadMe

The data for the Drimmel et al. (2003) map is installed in the code directory, because it is not very large.

Usage

All of the maps can be initialized similar to

import mwdust
drimmel= mwdust.Drimmel03(filter='2MASS H')
combined= mwdust.Combined15(filter='2MASS H')
combined19= mwdust.Combined19(filter='2MASS H')
sfd= mwdust.SFD(filter='2MASS H')

which sets up the Drimmel et al. (2003) map, the combined Bovy et al. (2016) map, an updated version of the combined map using the Green et al. (2019) Bayestar19 map, and the SFD map for the H-band filter. The maps can be evaluate for a given Galactic longitude l, Galactic latitude b, and an array (or scalar) of distances D

drimmel(60.,0.,3.) # inputs are (l,b,D)
array([ 0.38813341])
combined(30.,3.,numpy.array([1.,2.,3.,10.]))
array([ 0.22304147,  0.55687252,  0.86694602,  1.18779507])
# SFD is just the constant SFD extinction
sfd(30.,3.,numpy.array([1.,2.,3.]))
array([ 1.19977335,  1.19977335,  1.19977335])

and they can be plotted as a function of distance at a given (l,b)

combined.plot(55.,0.5) # inputs are (l,b)

(plot not shown). Maps that are derived from the HierarchicalHealpixMap.py class (currently all Green-type maps and the combined maps) can be vectorized to evaluate on array inputs of l, b, D

combined(numpy.array([30.,40.,50.,60.]),numpy.array([3.,4.,3.,6.]),numpy.array([1.,2.,3.,10.]))
array([0.22304147, 0.3780736 , 0.42528571, 0.22258065])

They can also be plotted on the sky using a Mollweide projection at a given distance using

combined.plot_mollweide(5.) # input is distance in kpc

Note that this requires healpy to be installed, so this does not work on Windows.

Supported bandpasses

Currently only a few filters are supported. To obtain E(B-V), specify filter='E(B-V)'. To check what bandpasses are supported on the sf10=True scale (these are all the bandpasses from Table 6 in Schlafly & Finkbeiner 2011), do

from mwdust.util import extCurves  
extCurves.avebvsf.keys()

which gives

['Stromgren u',
   'Stromgren v',
   'ACS clear',
   'CTIO R',
   'CTIO V',
   'CTIO U',
   'CTIO I',
   ...]

To check the bandpasses that are supported on the old SFD scale (sf10=False), do

numpy.array(extCurves.avebv.keys())[True-numpy.isnan(extCurves.avebv.values())]

which gives

array(['CTIO R', 'CTIO V', 'CTIO U', 'CTIO I', 'CTIO B', 'DSS-II i',
   'DSS-II g', 'WISE-1', 'WISE-2', 'DSS-II r', 'UKIRT H', 'UKIRT J',
   'UKIRT K', 'IRAC-1', 'IRAC-2', 'IRAC-3', 'IRAC-4', '2MASS H',
   'SDSS r', 'SDSS u', 'SDSS z', 'SDSS g', 'SDSS i', '2MASS Ks',
   '2MASS J'], dtype='|S14')

If no filter is supplied, E(B-V) is returned on the SFD scale if the object is initialized with sf10=True (which tells the code to use re-scalings from Schlafly & Finkbeiner 2011). sf10=True is the default initialization for every map, so be careful in interpreting the raw E(B-V) that come out of the code when not setting filter or when setting filter=None. Only use sf10=False when you have an extinction map in true E(B-V), not SFD E(B-V). No map currently included in this package is in this situation, so using sf10=False is never recommended.

Acknowledging mwdust and its data

When making use of this code in a publication, please cite Bovy et al. (2015a). Also cite the relevant papers for the dust map that you use:

mwdust's People

Contributors

dependabot[bot] avatar henrysky avatar jobovy avatar mileslucas avatar surhudm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

mwdust's Issues

Installation on mac

Hi - I'm using Python 3 installed with miniconda on a mac (osx 10.13.6). I am unable to install mwdust as I get the following error:

> sudo -E python3 setup.py install
...
/private/var/folders/tl/_5z6xmqd5_x4y0ljl2txp3740000gn/T/easy_install-cqf3s9qs/healpy-1.12.4/healpixsubmodule/src/cxx/autotools/cxxsupport/datatypes.h:74:10: fatal error:
'cstdint' file not found
#include
...

Google indicates that the problem may be some Clang/GCC issue - my Python installation uses GCC4.2.1, which it seems doesn't include cstdint. For example if I open Python3 it says:

Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 12:04:33)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin

My Xcode is up to date. Attached is the full installation report.

Cheers,

Martin
mwdust_install.txt

installation fails with Invalid cross-device link error

If DUST_DIR and git-cloned "mwdust" are on different partitions, installation by
cd mwdust
python setup.py install
fails reporting:

[...]
Downloading Sale et al. (2014) dust maps ...
[...]
2018-09-15 11:09:54 (9.21 MB/s) - ‘/home/[...]/MWDUST/sale14/Amap.tar.gz’ saved [111829141/111829141]

Amap.dat
ReadMe
Traceback (most recent call last):
File "setup.py", line 265, in
'Amap.dat'))
OSError: [Errno 18] Invalid cross-device link

Python 2.7.15rc1 on Ubuntu 18.04 machine.
regards, Michal

New dust map by Skowron+ 21

Hi,

I'd like to propose a new dust map to be added to the list of mwdust maps. Skowron+ 21 published a dust map in the direction of Magellanic Clouds based on OGLE red clump stars. The map is interactively accessable via their website. The reason why I'd really appreciate to implement this dust map is that the existing maps overestimate the extinction in the direction of MCs. This can be seen if one compares the Skowron's map to SFD on the linked website. Our group just recently published a paper (Molnár+ 21), where we also have addressed this issue (see Sect 3.2.1. and Fig. 4.).

Installing on Windows?

I'm trying to get mwdust installed on my Windows machine (so I can install isoclassify, so I can run a teammate's code) and am running into a snag on compiling the SFD C code. There are a couple of Unix-specific parts of the code; one of them is including unistd.h, which I found a workaround patch for, but Microsoft Visual C does not like 'm', and the StackOverflow advice to either remove that reference or replace it with 'msvcrt' has not worked.

Anything else I might try, or is this a sign to get my Linux box up and running again? :)

Thanks,
Jim

Dust Map Installation Not Working

I am a second-year undergraduate currently working on a project that involves the installation of the mwdust package. The installation commands are intended for Unix, yet I am trying to run them on a Windows machine. When I run setup.py, I get the following error message: "FileNotFoundError: [WinError 2] The system cannot find the file specified". Is there another way to install mwdust?

raise IOError('SFD/C module not found')

I am using Python 3.6.3 :: Anaconda custom (64-bit).I have installed mwdust using __" sudo python3 setup.py install --no-downloads"__
The dust data is also downloaded in the DUST_DIR.On importing mwdust I am getting the following error:
Traceback (most recent call last):
File "", line 1, in
File "/home/akriti/dust/mwdust-master/mwdust/init.py", line 1, in
from mwdust.SFD import SFD
File "/home/akriti/dust/mwdust-master/mwdust/SFD.py", line 7, in
from mwdust.util.read_SFD import read_SFD_EBV
File "/home/akriti/dust/mwdust-master/mwdust/util/read_SFD.py", line 27, in
raise IOError('SFD/C module not found')
OSError: SFD/C module not found
Also, if I close the terminal and open a new one i need to run setup.py again otherswise it says:
ModuleNotFoundError: No module named 'mwdust'

Cannot install with pip

I was trying to install mwdust with pip, but it failed with an error that says it can't find a C header file. After cloning the repo and using pip install . it worked. There might be a problem with the path of included directories.

Install: SUDO_USER environment variable doesnt exist

I've included the bash history of my attempt to install via instructions in the README, however I'm running into a problem: the environment variable SUDO_USER does not exist on my machine, but USER does. Simply replacing all instances of SUDO_USER with USER in setup.py fixes the problem for me. It may be useful to have setup.py check whether or not SUDO_USER is a valid environment variable and either throw an exception specific to that problem.

System:

  • Device : MacBook Pro (15-inch, Mid 2012)
  • OS: macOS sierra
  • CPU : 2.6 GHz Intel Core i7

Log:

$ python --version
Python 2.7.12 :: Anaconda 4.2.0 (x86_64)
$ git clone https://github.com/jobovy/mwdust.git
Cloning into 'mwdust'...
remote: Counting objects: 432, done.
remote: Total 432 (delta 0), reused 0 (delta 0), pack-reused 432
Receiving objects: 100% (432/432), 97.67 KiB | 0 bytes/s, done.
Resolving deltas: 100% (252/252), done.
$ cd mwdust
$ mkdir DUST_DIR
$ export pathname="${PWD}/DUST_DIR"
$ echo $pathname
/Users/jah5/Documents/installs/mwdust/DUST_DIR
$ echo "export DUST_DIR=\"${pathname}\"" >> ~/.bashrc
$ . ~/.bashrc
$ echo $DUST_DIR
/Users/jah5/Documents/installs/mwdust/DUST_DIR
$ mkdir ../mwdust_install
$ python setup.py install --prefix=../mwdust_install
Traceback (most recent call last):
  File "setup.py", line 51, in <module>
    'maps')])
  File "/Users/jah5/anaconda2/lib/python2.7/subprocess.py", line 536, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/Users/jah5/anaconda2/lib/python2.7/subprocess.py", line 523, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/Users/jah5/anaconda2/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/Users/jah5/anaconda2/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
TypeError: execv() arg 2 must contain only strings

In addition, I found that by manually setting the installation directory a la python setup.py install --prefix=... causes Python to complain that:

running install
Checking .pth file support in ../mwdust_install/lib/python2.7/site-packages/
/Users/jah5/anaconda2/bin/python -E -c pass
TEST FAILED: ../mwdust_install/lib/python2.7/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    ../mwdust_install/lib/python2.7/site-packages/

and your PYTHONPATH environment variable currently contains:

    ''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations


Please make the appropriate changes for your system and try again.

I may be misusing the --prefix option, but I found that simply installing as usual (python setup.py install) worked without a problem, and on systems where permissions are an issue, I believe that specifying the --user option (python setup.py install --user) should avoid the problems I ran into when using the --prefix option.

Error when opening combined data

Hi there,

I get the following error and traceback when I try to call in mwdust.Combined15:

---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-85-dd23bc76c50e> in <module>()
----> 1 combine = mwdust.Combined15()

/pkg/linux/anaconda2/lib/python2.7/site-packages/mwdust-1.0-py2.7-linux-x86_64.egg/mwdust/Combined15.pyc in __init__(self, filter, sf10, load_samples, interpk)
     32         HierarchicalHealpixMap.__init__(self,filter=filter,sf10=sf10)
     33         #Read the map
---> 34         with h5py.File(os.path.join(_combineddir,'dust-map-3d.h5'),'r') \
     35                 as combineddata:
     36             self._pix_info= combineddata['/pixel_info'][:]

/pkg/linux/anaconda2/lib/python2.7/site-packages/h5py/_hl/files.pyc in __init__(self, name, mode, driver, libver, userblock_size, swmr, **kwds)
    310             with phil:
    311                 fapl = make_fapl(driver, libver, **kwds)
--> 312                 fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
    313 
    314                 if swmr_support:

/pkg/linux/anaconda2/lib/python2.7/site-packages/h5py/_hl/files.pyc in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    140         if swmr and swmr_support:
    141             flags |= h5f.ACC_SWMR_READ
--> 142         fid = h5f.open(name, flags, fapl=fapl)
    143     elif mode == 'r+':
    144         fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5f.pyx in h5py.h5f.open()

IOError: Unable to open file (truncated file: eof = 229863588, sblock->base_addr = 0, stored_eof = 585045564)

Would this be an issue with the data file itself, or an H5Py issue?

QUESTION about mwdust.Combined19

Hi all,
I would like to use this package inside one of my code for calculating the extinction of a number of stars distributed in all sky.
My preference is to use the maps from mwdust.Green19, however they do not include some part of the sky.
I would like to ask what is the footprint of mwdust.Combined19, does it cover all the sky ? My feeling is that it will have the same footprint of mwdust.Green19.
If this is the case, there is some alternative map that cover all sky using as primary source mwdust.Green19, and that fill the missing parts with other maps ?
Thanks in advance

Issues with installation

Hello, I am running into issues with installation of mwdust. Running the setup.py script for mwdust gives:
running bdist_egg
running egg_info
writing mwdust.egg-info/PKG-INFO
writing dependency_links to mwdust.egg-info/dependency_links.txt
writing requirements to mwdust.egg-info/requires.txt
writing top-level names to mwdust.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
adding license file 'AUTHORS.txt'
writing manifest file 'mwdust.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.9-x86_64/egg
running install_lib
running build_py
running build_ext
building 'sfd_c' extension

After the last step listed, I got a long error attached to this message. I am running it on an M1 mac, but I also tested it on an intel chip mac to troubleshoot. Do you have any insight into this issue?

isoclassify error.txt

SFD vs. true

Hi Jo,

I had two questions about the scale on which E(B-V) is output: true (S&F 2011) vs. SFD (1998). I strongly suspect that I'm misunderstanding something; thanks in advance for your patience.

(1) What do I need to do to obtain E(B-V)true from this code?

For the Marshall and Drimmel maps, my understanding is that sf10=True means that E(B-V)true is rescaled in the code from E(B-V)true back to E(B-V)SFD, by dividing E(B-V)true by 0.86, because the S&F 2011 Table 6 is supposed to be applied to the old E(B-V)SFD values in order to get the new true filter extinctions. So if what I actually want to get out of the code is E(B-V)true, I either need to set "filter=None, sf10=False", or set "filter=None, sf10=True" and multiply the results by 0.86. Meanwhile, for the Green 2015 map, the output of "filter=None" is E(B-V)SFD no matter what I set sf10 to, so I'd always need to multiply the results by 0.86 to get E(B-V)true.

Is that right? It's a little confusing to me, because the documentation says "Only use sf10=False when you have an extinction map in true E(B-V), not SFD E(B-V)." When in fact the purpose of sf10=True is to convert from E(B-V)true to E(B-V)SFD, not the other way around; the issue in Marshall and Drimmel isn't that the map is in SFD, but that S&F Table 6 requires SFD inputs.

(2) Is the Drimmel extinction being overestimated, since it was already normalized to SFD and the code is scaling it up to SFD again?

The Drimmel AV map, in its initial construction by Drimmel et al. 2003, was already normalized to SFD. Doesn't that mean that AV,Drimmel = 3.09 * E(B-V)SFD
--> E(B-V)SFD = AV,Drimmel / 3.09,
which is already what we need, because S&F 2011 Table 6 takes SFD inputs. So why apply the 1/((1-self._sf10)+self._sf10*0.86) factor in Drimmel03.py? That gives the meaningless value of (1/0.86)*E(B-V)SFD or (1/0.86)2*E(B-V)true, and the extinction will be overestimated. Or did you somewhere scale AV,Drimmel to E(B-V)true and I didn't see it?

Anyway, I'm pretty sure I'm wrong, not the code, but I'd be very grateful for any help you can offer in explaining why I'm wrong.

Thanks again,
Adrian

DUST_DIR path length

For a sufficiently long DUST_DIR path length, the setup.py script fails. I don't have the terminal error. The path length which failed was
/Users/homefoldername/Documents/GitHub/Research/externaldata/DUST_DIR
changing the path to
/Users/homefoldername/Documents/DUST_DIR did install correctly.

Regarding environment variable in MWDust code.....

Dear all,
I am Rupak Roy a researcher in Astronomy, and interested to run the code MWDust. However, getting problem while installing. It will be nice if someone can help me in this regard. The detail is as given below :
I am using Python 2.7.13 :: Anaconda custom (64-bit).
Suppose my mwdust directory is at /home/rroy/Software/MWdust/mwdust-1.0
I made a directory at /home/rroy/Software/MWdust/DUST_DIR
I added this line in the .bashrc file in my home directory : export DUST_DIR="/home/rroy/Software/MWdust/DUST_DIR"
I sourced .bashrc and changed the directory to /home/rroy/Software/MWdust/mwdust-1.0
I ran the command : python setup.py install
I got the following python error :
Traceback (most recent call last):
File "setup.py", line 51, in
'maps')])
File "/home/rroy/Software/ANACONDA/anaconda2/lib/python2.7/subprocess.py", line 181, in check_call
retcode = call(*popenargs, **kwargs)
File "/home/rroy/Software/ANACONDA/anaconda2/lib/python2.7/subprocess.py", line 168, in call
return Popen(*popenargs, **kwargs).wait()
File "/home/rroy/Software/ANACONDA/anaconda2/lib/python2.7/subprocess.py", line 390, in init
errread, errwrite)
File "/home/rroy/Software/ANACONDA/anaconda2/lib/python2.7/subprocess.py", line 1024, in _execute_child
raise child_exception
TypeError: execv() arg 2 must contain only strings

I checked that before giving the error, the code generated an empty directory /home/rroy/Software/MWdust/DUST_DIR/maps

Looking forward for the suggestions..

Kind regards
Rupak Roy

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.