astropy/astropy-helpers

Helpers for Astropy and Affiliated packages

Astropy

The Astropy Project (https://astropy.org/) is a community effort to develop a single core package for Astronomy in Python and foster interoperability between Python astronomy packages. This repository contains the core package which is intended to contain much of the core functionality and some common tools needed for performing astronomy and astrophysics with Python.

Table of Contents

• Installation
• Contributing
• Getting Started with GitHub Codespaces
• Supporting the Project
• License

Installation

Releases are registered on PyPI, and development is occurring at the project's GitHub page.

For detailed installation instructions, see the online documentation or docs/install.rst in this source distribution.

To install astropy from PyPI, use:

pip install astropy

Contributing

The Astropy Project is made both by and for its users, so we welcome and encourage contributions of many kinds. Our goal is to keep this a positive, inclusive, successful, and growing community by abiding with the Astropy Community Code of Conduct.

More detailed information on contributing to the project or submitting feedback can be found on the contributions page. A summary of contribution guidelines can also be used as a quick reference when you are ready to start writing or validating code for submission.

Getting Started with GitHub Codespaces

Codespaces is a cloud development environment supported by GitHub. None of the Astropy build machinery depends on it, but it is a convenient way to quickly get started doing development on Astropy.

To get started, create a codespace for this repository by clicking this:

A codespace will open in a web-based version of Visual Studio Code. The dev container is fully configured with software needed for this project. For help, see the GitHub Codespaces Support page.

Note: Dev containers is an open spec which is supported by GitHub Codespaces and other tools.

Supporting the Project

The Astropy Project is sponsored by NumFOCUS, a 501(c)(3) nonprofit in the United States. You can donate to the project by using the link above, and this donation will support our mission to promote sustainable, high-level code base for the astronomy community, open code development, educational materials, and reproducible scientific research.

License

Astropy is licensed under a 3-clause BSD style license - see the LICENSE.rst file.

astropy-helpers's Issues

Build Cython C sources as prereq of sdist

Somehow I was convinced this was already done, to ensure that C files compiled from Cython sources are included in the sdist. But apparently that isn't working. Something to do.

packagename.__version__ has wrong git info when run in git repository

To reproduce, cd into any git repository and import a different astropy affiliated package. If the affiliated package is on a development version, packagename.__version__ will be "MAJOR.MINOR.devNUM" where "NUM" is wrong: it is the number of commits in the current git repository rather than that of packagename.

1.0

Python 3 fixes missing from 0.4.x release branch

@embray - some of the Python 3 fixes to e.g. viewcode.py are missing from the 0.4.x branch - is this deliberate?

Sphinx documentation should include module constants that have docstrings

This apparently used to work, but seems to have stopped. See astropy/astropy#3208 (comment) (refiled from astropy/astropy#3226 per @embray's comment).

smart_resolver isn't working for descriptor classes

For classes such as ConfigItem, which use the InheritDocstrings meta-class, the following code in smart_resolver doesn't work:

    if what in ('class', 'exception'):

because what ends up being attribute for those classes. I haven't figured out why yet, but it seems to be to do with the fact that type(ConfigItem) is not type.

Weird Astropy install errors on Python 2.6

I have a Linux machine with Python 2.6 and get this error when I try to run python setup.py test in the astropy git repo:

lfs1> time python setup.py test
Traceback (most recent call last):
  File "setup.py", line 41, in <module>
    cmdclassd = register_commands(NAME, VERSION, RELEASE)
  File "/nfs/us0/hfm/deil/code/astropy/astropy_helpers/astropy_helpers/setup_helpers.py", line 441, in register_commands
    add_command_option('install', *option)
  File "/nfs/us0/hfm/deil/code/astropy/astropy_helpers/astropy_helpers/setup_helpers.py", line 680, in add_command_option
    log.warning('Overriding existing {0!r} option '
AttributeError: 'module' object has no attribute 'warning'

real    0m3.648s
user    0m0.666s
sys 0m0.331s

If I run the same command again I get this different error:

lfs1> time python setup.py test
Freezing version number to astropy/version.py
Traceback (most recent call last):
  File "setup.py", line 112, in <module>
    **package_info
  File "/usr/lib64/python2.6/distutils/core.py", line 113, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python2.6/site-packages/setuptools/dist.py", line 221, in __init__
    self.fetch_build_eggs(attrs.pop('setup_requires'))
  File "/usr/lib/python2.6/site-packages/setuptools/dist.py", line 245, in fetch_build_eggs
    parse_requirements(requires), installer=self.fetch_build_egg
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 550, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (numpy 1.4.1 (/usr/lib64/python2.6/site-packages), Requirement.parse('numpy>=1.5.1'))

real    0m2.612s
user    0m1.107s
sys 0m0.342s

But actually I have numpy 1.7 installed, so the last error shouldn't have ocurred!?

lfs1> python
Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.7.0'

@embray @astrofrog Are these astropy or astropy-helpers issues or problems with my machine?

Validate that the package's version string is valid

astropy-helpers should do something to confirm that the package's version is valid (preferably conformant to PEP 440, but at least parseable if nothing else.

This came up because I had a test package in which the version was accidentally a float instead of a string and it led to all sorts of hard to debug badness :)

Add build_docs --args option

This is a copy of astropy/astropy#2906, which should have been filed as an astropy-helpers issue.

The python setup.py test command has a --args option to pass options to pytest.
It would be nice to have the same possibility to pass --args to sphinx-build via python setup.py build_sphinx.

For current help output of these two commands see here

Add --help support for command-line options supported by ah_bootstrap

As suggested in #27, I guess it would be good if ah_bootstrap.py integrated somewhat with distutils' command-line parsing so that, for example, command-line options used by ah_bootstrap can be reported to the user when they run setup.py --help.

build_sphinx -w fails with Python 3

In Astropy core, with Python 3:

$ python setup.py build_sphinx -w
Freezing version number to astropy/version.py
running build_sphinx
creating /Users/tom/Dropbox/Code/development/Astropy/astropy/docs/_build
creating /Users/tom/Dropbox/Code/development/Astropy/astropy/docs/_build/doctrees
creating /Users/tom/Dropbox/Code/development/Astropy/astropy/docs/_build/html
running build
running build_py
copying astropy/__init__.py -> build/lib.macosx-10.8-x86_64-3.4/astropy
copying astropy/version.py -> build/lib.macosx-10.8-x86_64-3.4/astropy
copying astropy/tests/helper.py -> build/lib.macosx-10.8-x86_64-3.4/astropy/tests
Fixing build/lib.macosx-10.8-x86_64-3.4/astropy/__init__.py build/lib.macosx-10.8-x86_64-3.4/astropy/version.py
Skipping implicit fixer: buffer
Skipping implicit fixer: idioms
Skipping implicit fixer: set_literal
Skipping implicit fixer: ws_comma
Fixing build/lib.macosx-10.8-x86_64-3.4/astropy/__init__.py build/lib.macosx-10.8-x86_64-3.4/astropy/version.py
Skipping implicit fixer: buffer
Skipping implicit fixer: idioms
Skipping implicit fixer: set_literal
Skipping implicit fixer: ws_comma
running build_ext
skipping 'astropy/table/_np_utils.c' Cython extension (up-to-date)
skipping 'astropy/time/erfa_time.c' Cython extension (up-to-date)
skipping 'astropy/io/ascii/cparser.c' Cython extension (up-to-date)
skipping 'astropy/convolution/boundary_extend.c' Cython extension (up-to-date)
skipping 'astropy/convolution/boundary_fill.c' Cython extension (up-to-date)
skipping 'astropy/convolution/boundary_none.c' Cython extension (up-to-date)
skipping 'astropy/convolution/boundary_wrap.c' Cython extension (up-to-date)
running build_scripts
Traceback (most recent call last):
  File "setup.py", line 112, in <module>
    **package_info
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/Users/tom/Dropbox/Code/development/Astropy/astropy/astropy_helpers/astropy_helpers/setup_helpers.py", line 922, in run
    stdo, stde = proc.communicate(subproccode)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 959, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 1601, in _communicate
    input_view = memoryview(self._input)
TypeError: memoryview: str object does not have the buffer interface

Add the possibility of adding a 2to3: False option to setup.cfg

It would be nice to be able to globally disable the use of 2to3 in a package rather than having to go through setup_package.py.

pip install of all affiliated package is broken with conda?

This is a continuation of astropy/astropy#2759 ... @astrofrog mentioned I should file an issue in the astropy-helpers tracker.

The problem that pip install fails for Astropy affiliated packages with an error message about the C compiler, even if the affiliated package is pure Python.

I tried ccdproc, spectral-cube and photutils ... here's the error message for spectral-cube:
https://gist.github.com/cdeil/06109508ce883a084d08,
https://gist.github.com/cdeil/276c3649890b298126ff

This should be easy to reproduce ... install Anaconda Python 3.4 on Ubuntu 12.04 or Mac OS X 10.9, then upgrade to Astropy 0.4, then run

pip install ccdproc

Having astropy-helpers breaks existing pip install

If I try and pip install a package that depends on astropy-helpers (but an older version than the released astropy-helpers) I get:

$ pip-3.3 install spectral_cube --user
Downloading/unpacking spectral-cube
  Downloading spectral-cube-0.1.tar.gz (153kB): 153kB downloaded
  Running setup.py egg_info for package spectral-cube
    Reading https://pypi.python.org/simple/astropy-helpers/
    Traceback (most recent call last):
      File "./ah_bootstrap.py", line 262, in _do_download
        dist = _Distribution(attrs=attrs)
      File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/setuptools/dist.py", line 239, in __init__
        self.fetch_build_eggs(attrs.pop('setup_requires'))
      File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/setuptools/dist.py", line 263, in fetch_build_eggs
        parse_requirements(requires), installer=self.fetch_build_egg
      File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pkg_resources.py", line 576, in resolve
        raise VersionConflict(dist,req) # XXX put more info here
    pkg_resources.VersionConflict: (astropy-helpers 0.4.dev (/private/var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/pip_build_tom/spectral-cube/astropy_helpers), Requirement.parse('astropy-helpers==0.4rc1'))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 16, in <module>
      File "/private/var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/pip_build_tom/spectral-cube/setup.py", line 8, in <module>
        import ah_bootstrap
      File "./ah_bootstrap.py", line 530, in <module>
        _main()
      File "./ah_bootstrap.py", line 527, in _main
        use_astropy_helpers(**kwargs)
      File "./ah_bootstrap.py", line 198, in use_astropy_helpers
        upgrade = _do_upgrade(dist, index_url)
      File "./ah_bootstrap.py", line 295, in _do_upgrade
        return _do_download(version=upgrade.version, index_url=index_url)
      File "./ah_bootstrap.py", line 277, in _do_download
        raise Exception(msg.format(source, repr(e)))
    Exception: Error retrieving astropy helpers from https://pypi.python.org/simple:
    VersionConflict(astropy-helpers 0.4.dev (/private/var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/pip_build_tom/spectral-cube/astropy_helpers), Requirement.parse('astropy-helpers==0.4rc1'))
    Complete output from command python setup.py egg_info:
    Reading https://pypi.python.org/simple/astropy-helpers/

Traceback (most recent call last):

  File "./ah_bootstrap.py", line 262, in _do_download

    dist = _Distribution(attrs=attrs)

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/setuptools/dist.py", line 239, in __init__

    self.fetch_build_eggs(attrs.pop('setup_requires'))

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/setuptools/dist.py", line 263, in fetch_build_eggs

    parse_requirements(requires), installer=self.fetch_build_egg

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pkg_resources.py", line 576, in resolve

    raise VersionConflict(dist,req) # XXX put more info here

pkg_resources.VersionConflict: (astropy-helpers 0.4.dev (/private/var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/pip_build_tom/spectral-cube/astropy_helpers), Requirement.parse('astropy-helpers==0.4rc1'))



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "<string>", line 16, in <module>

  File "/private/var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/pip_build_tom/spectral-cube/setup.py", line 8, in <module>

    import ah_bootstrap

  File "./ah_bootstrap.py", line 530, in <module>

    _main()

  File "./ah_bootstrap.py", line 527, in _main

    use_astropy_helpers(**kwargs)

  File "./ah_bootstrap.py", line 198, in use_astropy_helpers

    upgrade = _do_upgrade(dist, index_url)

  File "./ah_bootstrap.py", line 295, in _do_upgrade

    return _do_download(version=upgrade.version, index_url=index_url)

  File "./ah_bootstrap.py", line 277, in _do_download

    raise Exception(msg.format(source, repr(e)))

Exception: Error retrieving astropy helpers from https://pypi.python.org/simple:

VersionConflict(astropy-helpers 0.4.dev (/private/var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/pip_build_tom/spectral-cube/astropy_helpers), Requirement.parse('astropy-helpers==0.4rc1'))

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /private/var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/pip_build_tom/spectral-cube
Storing complete log in /Volumes/Raptor/.pip/pip.log

@embray - any idea what is happening here?

Sporadic failure in test

The following test fails from time to time:

====================================================== FAILURES =======================================================
_______________________________________________ test_no_cython_buildext _______________________________________________

tmpdir = local('/private/var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/pytest-39/test_no_cython_buildext0')

    def test_no_cython_buildext(tmpdir):
        """
        Regression test for https://github.com/astropy/astropy-helpers/pull/35

        This tests the custom build_ext command installed by astropy_helpers when
        used with a project that has no Cython extensions (but does have one or
        more normal C extensions).
        """

        # In order for this test to test the correct code path we need to fool
        # setup_helpers into thinking we don't have Cython installed
        setup_helpers._module_state['have_cython'] = False

        test_pkg = tmpdir.mkdir('test_pkg')
        test_pkg.mkdir('_eva_').ensure('__init__.py')

        # TODO: It might be later worth making this particular test package into a
        # reusable fixture for other build_ext tests

        # A minimal C extension for testing
        test_pkg.join('_eva_').join('unit01.c').write(dedent("""\
            #include <Python.h>
            #ifndef PY3K
            #if PY_MAJOR_VERSION >= 3
            #define PY3K 1
            #else
            #define PY3K 0
            #endif
            #endif

            #if PY3K
            static struct PyModuleDef moduledef = {
                PyModuleDef_HEAD_INIT,
                "unit01",
                NULL,
                -1,
                NULL
            };
            PyMODINIT_FUNC
            PyInit_unit01(void) {
                return PyModule_Create(&moduledef);
            }
            #else
            PyMODINIT_FUNC
            initunit01(void) {
                Py_InitModule3("unit01", NULL, NULL);
            }
            #endif
        """))

        test_pkg.join('setup.py').write(dedent("""\
            from os.path import join
            from setuptools import setup, Extension
            from astropy_helpers.setup_helpers import register_commands

            NAME = '_eva_'
            VERSION = 0.1
            RELEASE = True

            cmdclassd = register_commands(NAME, VERSION, RELEASE)

            setup(
                name=NAME,
                version=VERSION,
                cmdclass=cmdclassd,
                ext_modules=[Extension('_eva_.unit01',
                                       [join('_eva_', 'unit01.c')])]
            )
        """))

        test_pkg.chdir()
        run_setup('setup.py', ['build_ext', '--inplace'])
        sys.path.insert(0, str(test_pkg))
        try:
>           import _eva_.unit01
E           ImportError: No module named '_eva_.unit01'

/Users/tom/Dropbox/Code/development/Astropy/astropy-helpers/astropy_helpers/tests/test_setup_helpers.py:110: ImportError
------------------------------------------------ Captured stdout call -------------------------------------------------
running build_ext
building '_eva_.unit01' extension
creating build
creating build/temp.macosx-10.8-x86_64-3.4
creating build/temp.macosx-10.8-x86_64-3.4/_eva_
/usr/bin/clang -Wno-unused-result -Werror=declaration-after-statement -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -pipe -Os -I/opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c _eva_/unit01.c -o build/temp.macosx-10.8-x86_64-3.4/_eva_/unit01.o
creating build/lib.macosx-10.8-x86_64-3.4
creating build/lib.macosx-10.8-x86_64-3.4/_eva_
/usr/bin/clang -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names build/temp.macosx-10.8-x86_64-3.4/_eva_/unit01.o -o build/lib.macosx-10.8-x86_64-3.4/_eva_/unit01.so
copying build/lib.macosx-10.8-x86_64-3.4/_eva_/unit01.so -> _eva_
=================================== 1 failed, 49 passed, 1 skipped in 19.17 seconds ===================================

I wonder if there is a race condition? (this is in Python 3)

Release first stable version of astropy-helpers

@embray @eteq - I think we should do this soon, possibly even today or tomorrow. I've milestoned all issues I think we might want to deal with for this as 0.1, though we may decide on a different version number for the first release. What do you think?

six is now a required dependency to build the docs?

As mentioned in #78, six is now a required dependency for astropy-helpers. Do we want to keep it that way, or do we want to try and write our own fixes for the couple of places where six is used?

Incorporate metadata handling into astropy_helpers

astropy/package-template#41 included some changes that have setup.py read a project's metadata from its setup.cfg file (somewhat a la d2to1). At some point the way this is handled could be simplified using additional helpers from astropy_helpers. That, or astropy_helpers could be refactored to rely on d2to1 or something like it entirely.

This should not be a priority for the initial version of astropy_helpers, which is intended to minimize differences with the old astropy.*_helpers.py modules. Something worth looking into for the next version though.

build_sphinx broken

@embray - I can't remember whether we said we should not open issues here or not, so happy to close and move this. But currently Sphinx is broken on Python 2 due to the latest Python 3 fixes. #3 fixes the Python 2/3 compatibility for numpydoc, but we need to fix the other modules.

VersionConflict when pip installing

On RTD, I've seen the following error - when RTD installs dependencies it does pip install -r requirements.txt and requirements for pyregion are:

numpy
matplotlib
Cython
astropy-helpers
astropy
wcsaxes

The last one is an affiliated package. RTD then crashes with the following error:

Downloading/unpacking wcsaxes (from -r docs/rtd-pip-requirements (line 6))
  Running setup.py (path:/var/build/user_builds/pyregion/envs/latest/build/wcsaxes/setup.py) egg_info for package wcsaxes
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/var/build/user_builds/pyregion/envs/latest/build/wcsaxes/setup.py", line 8, in <module>
        import ah_bootstrap
      File "ah_bootstrap.py", line 530, in <module>
        _main()
      File "ah_bootstrap.py", line 527, in _main
        use_astropy_helpers(**kwargs)
      File "ah_bootstrap.py", line 198, in use_astropy_helpers
        upgrade = _do_upgrade(dist, index_url)
      File "ah_bootstrap.py", line 295, in _do_upgrade
        return _do_download(version=upgrade.version, index_url=index_url)
      File "ah_bootstrap.py", line 277, in _do_download
        raise Exception(msg.format(source, repr(e)))
    Exception: Error retrieving astropy helpers from https://pypi.python.org/simple:
    VersionConflict(astropy-helpers 0.4rc1.dev150 (/var/build/user_builds/pyregion/envs/latest/build/wcsaxes/astropy_helpers), Requirement.parse('astropy-helpers==0.4.3'))

I cannot reproduce this locally. @embray, do you have any idea what might be going on? The latest release of WCSAxes uses an older version of astropy-helpers than the one astropy uses but I wouldn't have thought that would matter?

Create changelog for astropy-helpers

I can do this later today.

Make it clearer/more discoverable how to use "--no-git"/"--offline"

While trying to do some astropy-helpers development, I noticed an oddity: Doing python setup.py --offline <whatever> does not stop astropy_helpers from getting updated from git. I think that's a bit counter-intuitive. I would have thought "offline" would mean don't get updates for a release and don't update git (after all, switching to the appropriate commit might require being online, right?). So is there any reason to not have the --offline option also imply --no-git? That is, it would then just act as a "use whatever I have in astropy_helpers right now, don't muck around with it."

An alternative is to introduce a third command that implies the two together.

Note that #31 may sort of "implicitly" resolve this by providing better docs, but it might be clearer to just do this anyway.

cc @embray

Sphinx oddity with doctests containing tracebacks

Yet another oddity with the docs that only seem to have recently shown up in Sphinx 1.2.3 (or possibly earlier, though not before Sphinx 1.2). See for example the last example on this page:

http://docs.astropy.org/en/stable/modeling/parameters.html#parameter-examples

Clearly the Traceback is shown twice, though the second time the exception message is also trucated to a single line. Very odd. Possibly a Sphinx bug more than it is something in our code but this is something to look into...

Attributes summary rendering problem with the Astropy autosummary customizations

When building the POPPY docs, which use the Astropy helpers, I noticed a weird rendering problem with the class attributes summary that I believe is due to customizations in the helpers.

To diagnose it, I edited astropy_helpers/sphinx/ext/astropyautosummary.py to change the name of get_items to bogus_get_items (making it functionally identical to Sphinx's built-in autosummary). This makes the output look like I expect it to.

@embray suggested I submit an issue (and ping @eteq )

``--no-git`` not working in astropy

If I do python setup.py build --no-git from astropy, I get the error "error: option --no-git not recognized". #27 was supposed to enable this, right, @embray ?

Tag v0.4.2 is not fetched by default.

Currently astropy is using astropy-helpers v0.4.2 (a93515f), but this tag is not referenced by a branch. So git fetch in an existing astropy-helpers clone will not fetch this tag. The issue is the same for git submodule update in a existing astropy clone, so it is not possible to update the submodule.
Currently the solution is to run git fetch --tags, but it would be much easier if the v0.4.2 tag was referenced by a branch.

astropy-helpers not reporting submodule actions

When I try e.g. building astropy, it does actions with git submodules (astropy_helpers, that is) completely silently. This leads to confusion about when/how the submodules actually are working, in contrast to the rest of the build process, which prints out a lot of information.

I see that there are supposed to be messages - ah_boostrap.py has bits like log.info('Updating submodule {0!r}'.format(submodule)). But they aren't appearing when I actually run the build process. I presume this is accidental? Is the solution just to put log.set_verbosity('info') at the top of ah_boostrap.py?

Installing astropy as root will introduce files owned by root in source directory

Here is an inconvenience that started when we introduced astropy-helpers:

I use my system python and my package manager in daily life, but I install astropy from source, such as sudo python setup.py install. When I do that, the install script pulls down the latest astropy-helpers and puts it in my source directory, but now those files are owned by "root".

In some cases, that means that I cannot do a "git pull" later: It will fail because it cannot overwrite certain files do to missing permissions. So, either I do

sudo python setup.py install
sudo chown -R moritz:moritz *

every time or I only touch the astropy sources when I am "root" (which I try to avoid in general).

We might have to live with this situation, since it mostly effects developers (who hopefully know how to deal with it) and only on certain system setups (and other people might think that I should not use such a setup in the first place, but rather switch to virtuelenv and friends), but I thought I bring it up here anyway.

Magically Appearing doc warnings

I am trying to make SunPy work with astropy_helpers and I am doing quite well. However 6 new doc warnings appear when I run sphinx-build using the astropy_helpers branch (https://github.com/Cadair/sunpy/tree/astropy_helpers) as opposed to the master branch (https://github.com/Cadair/sunpy/tree/master).

The errors are:

building [html]: targets for 47 source files that are out of date
updating environment: 268 added, 0 changed, 0 removed
WARNING: Didn't find sunpy.database.EntryAlreadyAddedError in sunpy.database  
WARNING: Didn't find sunpy.database.EntryAlreadyStarredError in sunpy.database
WARNING: Didn't find sunpy.database.EntryAlreadyUnstarredError in sunpy.database
WARNING: Didn't find sunpy.database.EntryNotFoundError in sunpy.database
WARNING: Didn't find sunpy.database.TagAlreadyAssignedError in sunpy.database
WARNING: Didn't find sunpy.database.NoSuchTagError in sunpy.database

These Exception classes are importable as sunpy.database.xyz and as I say with everything else the same switching on and off astropy_helpers triggers this.

Unexpected return value from locale.getdefaultlocale() causes failed pip install

This is on behalf of a former CfA colleague who is having trouble with pip install of APLpy into an anaconda environment. It appears that the following is returning None when a string is expected. I'm filing this in astropy-helpers, but if this is really an APLpy issue then let me know.

stdio_encoding = locale.getdefaultlocale()[1]

The full error log from pip is here:

------------------------------------------------------------
/Users/henghao/anaconda/bin/pip run on Tue Jul 22 08:54:06 2014
Downloading/unpacking aplpy
  Getting page https://pypi.python.org/simple/aplpy/
  URLs to search for versions for aplpy:
  * https://pypi.python.org/simple/aplpy/
  Analyzing links from page https://pypi.python.org/simple/APLpy/
    Found link https://pypi.python.org/packages/source/A/APLpy/APLpy-0.9.10.tar.gz#md5=049317adf7a08ede6e7e0d5c7e18d722 (from https://pypi.python.org/simple/APLpy/), version: 0.9.10
    Found link https://pypi.python.org/packages/source/A/APLpy/APLpy-0.9.11.tar.gz#md5=53a2130136dd6e0b2fbf1c8e8b15d3e6 (from https://pypi.python.org/simple/APLpy/), version: 0.9.11
    Found link https://pypi.python.org/packages/source/A/APLpy/APLpy-0.9.12.tar.gz#md5=aeff0c49f43847819ab5e94e63abd74c (from https://pypi.python.org/simple/APLpy/), version: 0.9.12
    Found link https://pypi.python.org/packages/source/A/APLpy/APLpy-0.9.5.tar.gz#md5=241951320d7ddcb7efff141f836ee061 (from https://pypi.python.org/simple/APLpy/), version: 0.9.5
    Found link https://pypi.python.org/packages/source/A/APLpy/APLpy-0.9.6.tar.gz#md5=bfd8e61ea1139dcc3d8bdf94eee03df3 (from https://pypi.python.org/simple/APLpy/), version: 0.9.6
    Found link https://pypi.python.org/packages/source/A/APLpy/APLpy-0.9.7.tar.gz#md5=4f667fdd71ae91e4c976f15bf43f7e97 (from https://pypi.python.org/simple/APLpy/), version: 0.9.7
    Found link https://pypi.python.org/packages/source/A/APLpy/APLpy-0.9.8.tar.gz#md5=a93848f72951e3114c5b7a618750ca82 (from https://pypi.python.org/simple/APLpy/), version: 0.9.8
    Found link https://pypi.python.org/packages/source/A/APLpy/APLpy-0.9.9.tar.gz#md5=10d247bdf42f920bc9ca01368a5e13b8 (from https://pypi.python.org/simple/APLpy/), version: 0.9.9
    Skipping link http://aplpy.github.com (from https://pypi.python.org/simple/APLpy/); not a file
    Skipping link http://aplpy.github.com/ (from https://pypi.python.org/simple/APLpy/); not a file
    Skipping link http://aplpy.github.io (from https://pypi.python.org/simple/APLpy/); not a file
    Skipping link http://aplpy.sourceforge.net (from https://pypi.python.org/simple/APLpy/); not a file
    Skipping http://cloud.github.com/downloads/aplpy/aplpy/APLpy-0.9.7.tar.gz (from https://pypi.python.org/simple/APLpy/) because it is externally hosted.
    Skipping http://cloud.github.com/downloads/aplpy/aplpy/APLpy-0.9.8.tar.gz (from https://pypi.python.org/simple/APLpy/) because it is externally hosted.
    Skipping http://voxel.dl.sourceforge.net/sourceforge/aplpy/APLpy-0.9.4.tar.gz (from https://pypi.python.org/simple/APLpy/) because it is externally hosted.
    Skipping https://github.com/downloads/aplpy/aplpy/APLpy-0.9.5.tar.gz (from https://pypi.python.org/simple/APLpy/) because it is externally hosted.
  Using version 0.9.12 (newest of versions: 0.9.12, 0.9.11, 0.9.10, 0.9.9, 0.9.8, 0.9.7, 0.9.6, 0.9.5)
  Downloading from URL https://pypi.python.org/packages/source/A/APLpy/APLpy-0.9.12.tar.gz#md5=aeff0c49f43847819ab5e94e63abd74c (from https://pypi.python.org/simple/APLpy/)
  Running setup.py (path:/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/setup.py) egg_info for package aplpy
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/setup.py", line 8, in <module>
        import ah_bootstrap
      File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/ah_bootstrap.py", line 716, in <module>
        _main()
      File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/ah_bootstrap.py", line 713, in _main
        use_astropy_helpers(**kwargs)
      File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/ah_bootstrap.py", line 202, in use_astropy_helpers
        is_submodule = _check_submodule(path, use_git=use_git)
      File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/ah_bootstrap.py", line 392, in _check_submodule
        return _check_submodule_using_git(path)
      File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/ah_bootstrap.py", line 435, in _check_submodule_using_git
        stderr = stderr.decode(stdio_encoding)
    TypeError: decode() argument 1 must be str, not None
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/setup.py", line 8, in <module>

    import ah_bootstrap

  File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/ah_bootstrap.py", line 716, in <module>

    _main()

  File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/ah_bootstrap.py", line 713, in _main

    use_astropy_helpers(**kwargs)

  File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/ah_bootstrap.py", line 202, in use_astropy_helpers

    is_submodule = _check_submodule(path, use_git=use_git)

  File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/ah_bootstrap.py", line 392, in _check_submodule

    return _check_submodule_using_git(path)

  File "/private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy/ah_bootstrap.py", line 435, in _check_submodule_using_git

    stderr = stderr.decode(stdio_encoding)

TypeError: decode() argument 1 must be str, not None

----------------------------------------
Cleaning up...
  Removing temporary dir /private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao...
Command python setup.py egg_info failed with error code 1 in /private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy
Exception information:
Traceback (most recent call last):
  File "/Users/henghao/anaconda/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/Users/henghao/anaconda/lib/python3.4/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/Users/henghao/anaconda/lib/python3.4/site-packages/pip/req.py", line 1229, in prepare_files
    req_to_install.run_egg_info()
  File "/Users/henghao/anaconda/lib/python3.4/site-packages/pip/req.py", line 325, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/Users/henghao/anaconda/lib/python3.4/site-packages/pip/util.py", line 697, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command python setup.py egg_info failed with error code 1 in /private/var/folders/rj/1_zbz2gd3_71h7m32_hjp1wm0000gn/T/pip_build_henghao/aplpy

Autodownload of astropy-helpers not working

The requested path 'astropy_helpers' for importing astropy_helpers does not exist, or does not contain a copy of the astropy_helpers pacakge.  Attempting download instead.
Downloading astropy_helpers; run setup.py with the --offline option to force offline installation.
Error retrieving astropy helpers from https://pypi.python.org/simple:
DistutilsError("Could not find suitable distribution for Requirement.parse('astropy-helpers')",)

If the problem persists consider installing astropy_helpers manually using pip
(`pip install astropy_helpers`) or by manually downloading the source archive,
extracting it, and installing by running `python setup.py install` from the
root of the extracted source code.
“

@ssim found this error on a clean mavericks machine.

Move test stuff to astropy-helpers if there's a need for it outside astropy-dependents

In APE4 and the current master, the test runner and related framework are not in astropy-helpers, but rather in astropy. The justification from @embray in APE4 is that the tests have to be at runtime, so it makes sense to leave it in astropy given that affiliated packages will need it at runtime anyway.

But while working on #72 I realized an issue with that: the test runner and test command are now in separate packages, so they can't be kept in sync. More generally, it's really confusing to figure out how they interplay when they are in different packages. So maybe we should just move all the testing stuff to astropy-helpers too?

cc @astrofrog

Add generic support for submodules

It came up in astropy/photutils#132 that it might be nice to be able to have git submodules available in an affiliated package. However, the bar for use of submodules is very high: for good or for ill, most people I talk about it are afraid of them/don't understand how to use them.

Fortunately, ah_bootstrap has solved this problem already, as has the ipython setup.py : just update the submodules as part of the setup process.It would be great to put machinery like this in the astropy-helpers, so that affiliated packages and astropy can start using it.

@embray, do you thing the ah_bootstrap code can be translated fairly easily into a setup.py command to do submodule updating (a la ipython)? Or would it be better to lift some of it from ipython, or start from scratch? (or I suppose "not do it at all" is another option)

Problem finding correct version of astropy_helpers (?)

I'm trying to install the development version following the instructions here. Things are failing at step 6 like so:

(astropy_dev_ENV)bash-3.2$ python setup.py develop
Traceback (most recent call last):
  File "setup.py", line 21, in <module>
    from astropy_helpers.distutils_helpers import is_distutils_display_option
ImportError: No module named distutils_helpers

I do have other versions of astropy_helpers installed -- could this be the problem? Thanks.

``setup.py`` fails if in a git repo with no commits (low priority)

The following demonstrates the issue:

$ git clone [email protected]:astropy/package-template.git
Cloning into 'package-template'...
cd packaremote: Counting objects: 545, done.
remote: Total 545 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (545/545), 160.67 KiB | 0 bytes/s, done.
Resolving deltas: 100% (273/273), done.
Checking connectivity... done.
mac-robitaille2:tmp tom$ cd package-template
mac-robitaille2:package-template tom$ rm -rf .git
mac-robitaille2:package-template tom$ git init

mac-robitaille2:package-template tom$ python setup.py
git submodule command failed unexpectedly:
error: pathspec 'astropy_helpers' did not match any file(s) known to git.
Did you forget to 'git add'?

The requested path 'astropy_helpers' for importing astropy_helpers does not exist, or does not contain a copy of the astropy_helpers package.  Attempting download instead.
Downloading astropy_helpers; run setup.py with the --offline option to force offline installation.
Traceback (most recent call last):
  File "setup.py", line 18, in <module>
    from astropy_helpers.setup_helpers import (
  File "/Users/tom/tmp/package-template/astropy_helpers-0.4.3-py3.4.egg/astropy_helpers/__init__.py", line 2, in <module>
    from .version import version as __version__
  File "/Users/tom/tmp/package-template/astropy_helpers-0.4.3-py3.4.egg/astropy_helpers/version.py", line 151, in <module>
    version = update_git_devstr(_last_generated_version)
  File "/Users/tom/tmp/package-template/astropy_helpers-0.4.3-py3.4.egg/astropy_helpers/version.py", line 34, in update_git_devstr
    devstr = get_git_devstr(sha=True, show_warning=False, path=path)
  File "/Users/tom/tmp/package-template/astropy_helpers-0.4.3-py3.4.egg/astropy_helpers/version.py", line 144, in get_git_devstr
    return _decode_stdio(stdout)[:40]
  File "/Users/tom/tmp/package-template/astropy_helpers-0.4.3-py3.4.egg/astropy_helpers/version.py", line 17, in _decode_stdio
    text = stream.decode(stdio_encoding)
AttributeError: 'str' object has no attribute 'decode'

While this may seem like a quite obscure case, it can happen when people are setting up affiliated packages from scratch (I ran into it).

Feature request: --use-ccache option

ccache can provide factors of several improvement in build time for astropy. Configuring ccache on every machine is a bit of a pain so I was thinking it could be nifty to have a new option --use-ccache and a corresponding use_ccache option in setup.cfg that could then indicate that we should make use of ccache wherever possible. Since we already have an adjust_compiler function it would be trivial to use this to prepend the compilation command with the ccache command.

@eteq @embray @mdboom - what do you think about this?

Inherited members always ignored in automodsumm

The sphinx extension automodsumm always ignores inherited class members when producing an auto summary. This is different from the default behavior of autosummary. I'd like to add a configuration variable to automodsumm to be able to revert this (perhaps automodsumm_inherited_members), but I wanted to bring it up here, as the behavior differs from autosummary, which I find to be interesting / a possible bug.

Add astropy doc build to travis?

While looking at #69, something occurred to me: a lot of the functionality here does not get tested, because it depends on getting used in some other package. Particularly some of the docs stuff is not exhaustively unit tested because it's implicitly tested when you try to use it.

So a possible solution would be to add an astropy sphinx build to the travis build of astropy-helpers. Of course that will take a lot longer, but it seems like the best way to actually test this stuff without just waiting to see if it kills astropy master.

@embray, what do you think?

package-template sub-module causing issues

At the moment, the package-template sub-module is causing issues with affiliated packages that try and pip install the latest version of astropy-dev:

Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Clone of '[email protected]:embray/package-template.git' into submodule path 'astropy_helpers/tests/package_template' failed
Failed to recurse into submodule path 'astropy_helpers/astropy_helpers/tests/package_template/astropy_helpers'
Failed to recurse into submodule path 'astropy_helpers/astropy_helpers/tests/package_template'

Failed to recurse into submodule path 'astropy_helpers'

This might go away once we merge in the changes to the package-template and change the sub-module URL, but either way it seems maybe over the top that cloning astropy-dev ultimately results in cloning the package template. I would be in favor of removing the package-template sub-module from astropy-helpers and only including it for example as part of a test, or in .travis.yml.

Catch KeyboardInterrupt and pass to pytest during test run

If you run a test session using the pytest CLI and you hit an import error or similar you can interrupt the test run and see the error traceback without having to wait for the whole test run to complete. Would it be possible to replicate this behaviour inside setup.py?

Support subpackage scripts

It should be possible for each subpackage to specify what console scripts they provide and the entry points (main functions) of those scripts. See astropy/astropy#3271

Confusing message from ah_bootstrap.py when installing from source dist when in a subdirectory of a git repo

The Astropy source distribution currently bundles astropy_helpers. When installing a fresh copy of Astropy from source this results in the messages:

Downloading/unpacking astropy
  Downloading astropy-0.4.2.tar.gz (7.2MB): 7.2MB downloaded
  Running setup.py (path:/internal/1/root/src/astropy/astropy/.tox/py33/build/astropy/setup.py) egg_info for package astropy
    git submodule command failed unexpectedly:
    error: pathspec '.tox/py33/build/astropy/astropy_helpers' did not match any file(s) known to git.
    Did you forget to 'git add'?


Requirement already satisfied (use --upgrade to upgrade): numpy>=1.5.1 in ./lib/python3.3/site-packages (from astropy)
Installing collected packages: astropy
  Running setup.py install for astropy
    git submodule command failed unexpectedly:
    error: pathspec '.tox/py33/build/astropy/astropy_helpers' did not match any file(s) known to git.
    Did you forget to 'git add'?

The error message appears twice--once when ./setup.py egg_info is run, and once when ./setup.py install is run. This is only happening because I was installing Astropy in a tox environment under the .tox directory in a project, which happens to live under a git repo. So this is somewhat related to #107 in that it has to do with astropy-helpers (in this case ah_bootstrap.py itself, I think) not correctly detecting whether it is in a git repo for the project being installed.

astropy-helpers's default is to use astropy's pytest

Is it possible to make the system py.test the default (looking in the code, that seems to already be the way, but it doesn't seem to work).

setup.cfg ah_bootstrap auto_use=False does not work completely

This is a copy of astropy/astropy#2941 since it seems to be a astropy-helpers problem:

When I set in setup.cfg of astropy (0.4.1)

[ah_bootstrap]
auto_use = False

and then try to build the package with an external astropy_helpers package (version 0.4.2), I get the error

Traceback (most recent call last):
  File "<stdin>", line 27, in <module>
  File "/usr/lib/python2.7/dist-packages/sphinx/application.py", line 147, in __init__
    self._init_builder(buildername)
  File "/usr/lib/python2.7/dist-packages/sphinx/application.py", line 209, in _init_builder
    self.emit('builder-inited')
  File "/usr/lib/python2.7/dist-packages/sphinx/application.py", line 408, in emit
    results.append(callback(self, *args))
  File "/tmp/buildd/python-astropy-0.4.1+dfsg/astropy_helpers/astropy_helpers/sphinx/ext/automodsumm.py", line 234, in process_automodsumm_generation
    base_path=app.srcdir)
  File "/tmp/buildd/python-astropy-0.4.1+dfsg/astropy_helpers/astropy_helpers/sphinx/ext/automodsumm.py", line 419, in generate_automodsumm_docs
    name, obj, parent = import_by_name(name)
ValueError: too many values to unpack
Sphinx Documentation subprocess failed with return code 1

Which shows that here the internal code is used instead of the external. When I remove the astropy_helpers subdirectory from the astropy package, everything works fine.

Fix warnings in setuptools 8.0

As of setuptools 8.0, which introduced some new version parsing and full support for PEP 440, we get a couple warnings when running setup.py:

    /home/iguananaut/.virtualenvs/tmp-dea251bdca0911e7/local/lib/python2.7/site-packages/pkg_resources/__init__.py:186: RuntimeWarning: You have iterated over the result of pkg_resources.parse_version. This is a legacy behavior which is inconsistent with the new version class introduced in setuptools 8.0. That class should be used directly instead of attempting to iterate over the result.
      stacklevel=1,
    /home/iguananaut/.virtualenvs/tmp-dea251bdca0911e7/local/lib/python2.7/site-packages/setuptools/dist.py:284: UserWarning: The version specified requires normalization, consider using '1.1.dev0' instead of '1.1.dev'.
      self.metadata.version,

I'm not sure what the first one is aobout, I'm sure we can figure it out. The second one is because we use "dev" version numbers like "1.1.dev". But if I understand correctly (and I need to double check this) PEP-440 requires that dev versions have some unique specifier following it and can't just be bare. We already generate a 'revision' number that we attach (like 1.1.dev1234) based on a count of git changesets. But this count (unlike the githash) does not get stored in version.py for use in source dists of dev versions. So that should be fixed as well.

Populate README/other documentation

I realize the priority now is getting things working, but I thought I would put this in as a reminder... Currently the README is empty, and there's not really any documentation in this repository about astropy-helpers, what it is, how it works, etc. That's definitely worth doing once APE4 is actually merged.

Make Sphinx extensions Python-3 compatible

This is done for numpydoc in #3 but the other extensions need to be done too.

Document "common commands"

I realise writing full documentation for astropy-helpers will take time and it not very urgent.

Would it be possible to start with a very short document of "common commands" to work with astropy-helpers? Here's some things I'd like to know:

• If I want to make a change to astropy-helpers, should I do that in the git submodule of astropy or check out a separate copy of the repo? In the past I've used the git submodule, but then after making my commit in a branch I didn't know the git commands to get back to a "clean state" locally.
• At my institute git via the HTTPS or GIT protocol doesn't work, only via SSH. If I clone astropy via SSH, the astropy-helpers submodule still has a HTTPS URL. Which command should I use to change this to an SSH URL (in the astropy or astropy-helpers repo?)
• Which git commands should be used to update the git submodule pointer to astropy-helpers after some fix has been made?

cc @embray @eteq

Decide on version numbering

Since we don't have to have a release schedule synchronized with astropy, what version numbering do we want to assume? Do we start at 0.1, 0.9, 0.4, 1.0? Or do we want the version number to have a reference to the current astropy version?

Consider renaming astropy-helpers in the long term

Glue (http://glueviz.org) is an example of a non-astro-specific package where I think it would be nice to make use of astropy-helpers. We had previously discussed that in the long term we might consider renaming astropy-helpers to something more generic so that it can be used by non-astro packages (of course it can be used already, but having 'astro' in the name can make it sound like it is astro-specific). So this issue is just to start a conversation about whether we should consider renaming astropy-helpers to something else in future? (e.g. setup-helpers?)