esheldon/esutil

A variety of python utilities focusing on numerical, scientific, and astrophysical computing

A python package including a wide variety of utilities, focused primarily on numerical python, statistics, and file input/output. Includes specialized tools for astronomers.

Installation

The easiest way is through conda which has pre-compiled binaries

conda install esutil

# or if you don't have conda forge in your channels
conda install -c conda-forge esutil

You can also use pip

pip install esutil

Windows

The unit tests pass on windows with the exception of the recfile package (and the sfile package that uses it).

esutil's Issues

coords.sphdist fails if ra2, dec2 is array but ra1, dec1 is scalar

numpy.where messes it up. you can't unpack it as done in coord.py:487 when the 
dimension of the input is messy. 

To achieve the same clipping, use numpy.clip:


    costheta.clip(min=-1.0, max=1.0, out=costheta)


It's faster and error proofer. There are several similar cases in the code.

Original issue reported on code.google.com by [email protected] on 7 Apr 2012 at 7:11

warnings for newer numpy versions

Looks like numpy is being persnickety.

/Users/Matt/miniconda3/envs/anl/lib/python3.6/site-packages/esutil/wcsutil.py:1201: DeprecationWarning: Non-string object detected for the array ordering. Please pass in 'C', 'F', 'A', or 'K' instead
    x = x.flatten(1)

Python 3 tag

@esheldon We are currently migrating the LSST stack to support Python 3. Since this is currently a third party dependency, we are very interested in having a tagged version that runs under Python 3. Is that in the short term plan? Is there anything I can do to help that happening?

lookup_id segfaults

I just installed this on top of a miniconda install on a Mac running El Cap. When trying to use the lookup_id method, it segfaults unless another method is called first.

E.g.

Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec  6 2015, 18:57:58)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import esutil
>>> htm = esutil.htm.HTM(8)
>>> htm.lookup_id(1., 2.)
Segmentation fault: 11

but

Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec  6 2015, 18:57:58)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import esutil
>>> htm = esutil.htm.HTM(8)
>>> htm.intersect(1.,2.,0.0001)
array([1016022])
>>> htm.lookup_id(3., 4.)
array([1016776])

I haven't look into it, but it seems like there is some side effect that is making an assignment lookup_id expects.

segfault if cosmology object is deepcopied

I suppose this is something that isn't supported, but should probably raise an exception rather than completely segfaulting...(tested on both linux and mac)

In [2]: import copy
In [3]: cosmo=esutil.cosmology.Cosmo()
In [4]: cosmo.Dl(0,1.0)
Out[4]: 4625.360378585448
In [5]: cosmo2=copy.deepcopy(cosmo)
In [6]: cosmo2.Dl(0,1.0)
Segmentation fault

`-std=c++11` breaks `clang` builds

The fix from #75 breaks the build process on Macs with clang (the default compiler), because clang doesn't allow the -std=c++11 flag when compiling C code. So ideally, you'd only want to have this flag on for the *.cpp files, but not for the *.c files. However, this seems quite difficult to do, as distutils tries hard to use the same flags for all files. The easiest solution might just be to make this an optional flag that can be turned off or detecting that this flag doesn't work and turning it off automatically. Note that the current way of checking the compiler flags doesn't detect this issue, because it compiles a C++ file and for that the flag is fine. You could do the check with a C file, but I'm not sure that that doesn't lead to other issues.

Method to get total number of HTM pixels

A useful addition would be a method to get the total number of HTM pixels for a given depth. I think it would look like this:

def get_n_pixels(self):
    """Return the total number of pixels for this HTM's depth."""
    return 4**self.get_depth()

but I had to work that out from the existing code. This is useful for knowing how many files you might generate with a given indexing scheme.

htm match expects ascii names for matchfile; fails silently for unicode names

What steps will reproduce the problem?

matchfile = u'filename'
h = eu.htm.HTM(8)
h.match(lenses['RA'], lenses['DEC'], shapes[config['shape_ra_key']], 
shapes[config['shape_dec_key']], maxrange, maxmatch=-1, file=matchfile)

What is the expected output? What do you see instead?

No matchfile is created, no error/warning message is throw. Unicode names are 
returned by some of python filesystem function, in this case glob, which cause 
the code to fail working properly.

What version of the product are you using? On what operating system?

esutil 0.5.1 on Linux with python 2.7.3

Original issue reported on code.google.com by [email protected] on 6 Dec 2014 at 6:39

deprecation warning

Looks like numpy is changing...

pizza_cutter/des_pizza_cutter/tests/test_sky_bounds.py::test_get_rough_sky_bounds_smoke
  /Users/Matt/miniconda3/envs/anl/lib/python3.6/site-packages/esutil/wcsutil.py:1201: DeprecationWarning: Non-string object detected for the array ordering. Please pass in 'C', 'F', 'A', or 'K' instead
    x = x.flatten(1)
  /Users/Matt/miniconda3/envs/anl/lib/python3.6/site-packages/esutil/wcsutil.py:1202: DeprecationWarning: Non-string object detected for the array ordering. Please pass in 'C', 'F', 'A', or 'K' instead
    y = y.flatten(1)

Problems installing esutil in Mac OS

I have found a problem installing esutil on a Mac. I run Mac OS 10.13.3. I get failures both using pip and when I try to install the tar file downloaded from gitHub. After some research in StackOverflow, I have found that some python packages can have similar issues when one of their dependencies is outdated, but as far as I know I have all the esutil dependencies up to date. I append the relevant parts of the error message below.

gcc -fno-strict-aliasing -I/Users/badenes/anaconda/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/badenes/anaconda/lib/python2.7/site-packages/numpy/core/include -Iesutil/include -Iesutil/recfile -Iesutil/htm -Iesutil/htm/htm_src -I/Users/badenes/anaconda/include/python2.7 -c esutil/recfile/records.cpp -o build/temp.macosx-10.6-x86_64-2.7/esutil/recfile/records.o -arch i386 -arch x86_64 cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ In file included from /Users/badenes/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1816:0, from /Users/badenes/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, from /Users/badenes/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4, from esutil/recfile/records.hpp:9, from esutil/recfile/records.cpp:1: /Users/badenes/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] #warning "Using deprecated NumPy API, disable it by " \ ^ /var/tmp//ccMCfGFR.s:62:bad register name `%rbp’
After which follow many similar bad register name warnings, and then

``
/var/tmp//ccMCfGFR.s:36484:Rest of line ignored. 1st junk character valued 64 (@).
error: command 'gcc' failed with exit status 1

Failed building wheel for esutil
``

Request for a direct cone search method.

Running cone searches on databases indexed on HTM IDs is a fairly common 
requirement, in my opinion. There appears to be no straightforward method in 
the HTM class which returns HTM IDs for a given RA, DEC and radius. It would be 
wonderful if some means were provided to achieve this.

Original issue reported on code.google.com by [email protected] on 25 Sep 2014 at 3:21

find=False option on sky2image broken

It appears the find=False option is broken in the latest esutil.

imp is deprecated

I've started getting deprecation warnings from things that import esutil that the imp module is deprecated.

$ python -Wd
Python 3.6.10 | packaged by conda-forge | (default, Mar  5 2020, 09:56:10) 
[GCC Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import esutil
/anaconda3/envs/py3/lib/python3.6/site-packages/esutil/htm/htmc.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp

I'm sure there is still lots of time before this stops working, but probably worth fixing. Looks pretty easy.

Installation issues on Windows

What steps will reproduce the problem?
1. python setup.py install

What is the expected output? What do you see instead?
Should not see any errors, instead:

running install
running build
running build_py
running build_ext
building 'esutil.recfile._records' extension
C:\MinGW\bin\gcc.exe -mdll -O -Wall 
-IC:\Python27\lib\site-packages\numpy\core\include -Iesutil/include 
-Iesutil/htm -Iesutil/htm/htm_src -IC:\Python27\include -IC:\Python27\PC -c 
esutil/recfile/records.cpp -o 
build\temp.win32-2.7\Release\esutil\recfile\records.o
In file included from esutil/recfile/records.cpp:1:0:
esutil/recfile/records.hpp:33:1: warning: missing terminating " character 
[enabled by default]
esutil/recfile/records.hpp:90:1: warning: missing terminating " character 
[enabled by default]
esutil/recfile/records.hpp:105:3: warning: missing terminating " character 
[enabled by default]
esutil/recfile/records.hpp:132:56: warning: missing terminating " character 
[enabled by default]
esutil/recfile/records.hpp:142:3: warning: missing terminating " character 
[enabled by default]
esutil/recfile/records.hpp:169:3: warning: missing terminating " character 
[enabled by default]
esutil/recfile/records.hpp:178:3: warning: missing terminating " character 
[enabled by default]
esutil/recfile/records.hpp:182:3: warning: missing terminating " character 
[enabled by default]
esutil/recfile/records.cpp: In member function 'void Records::DoSeek(npy_intp)':
esutil/recfile/records.cpp:321:43: error: 'fseeko' was not declared in this 
scope
esutil/recfile/records.cpp: In member function 'void 
Records::SkipFieldAsBinary(long long int)':
esutil/recfile/records.cpp:475:41: error: 'fseeko' was not declared in this 
scope
esutil/recfile/records.cpp: In member function 'void 
Records::SkipBinaryRows(long long int)':
esutil/recfile/records.cpp:539:45: error: 'fseeko' was not declared in this 
scope
error: command 'gcc' failed with exit status 1


What version of the product are you using? On what operating system?
This is esutil-v0.5.1 using Windows 7 64 bit and Python 2.7.

Please provide any additional information below.
Thank you very much for any help or assistance you can provide!

Original issue reported on code.google.com by [email protected] on 7 Mar 2013 at 6:59

remove stomp_util.py

The stomp package is abandoned, no point in supporting it. And it conflicts with a different stomp, see #60

Reverse indices are not properly sorted (and can change from version to version)

The following code snippet (a) doesn't return sorted indices and (b) results in different ordering from numpy 1.24 to numpy 1.26 on x86 processors.

import numpy as np
import esutil


arr = np.zeros(1000, dtype=np.int64)
arr[500: ] = 1

h, rev = esutil.stat.histogram(arr, rev=True)

print(rev[rev[0]: rev[1]])
print(rev[rev[1]: rev[2]])

The fix is that the histogram call to argsort should use kind="stable" which properly preserves sort order.

I don't want to fix this yet to allow my code to properly prepare for the different (but correct!) ordering.

Problems installing esutil on Mac OS 2

Hello,

I have tried to install esutil on my Mac (10.14.2) and the same error message keeps coming up when it starts to build the module,

building 'esutil.recfile._records' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/anaconda3/include -I/anaconda3/include -I/anaconda3/lib/python3.7/site-packages/numpy/core/include -Iesutil/include -Iesutil/recfile -Iesutil/htm -Iesutil/htm/htm_src -I/anaconda3/include/python3.7m -c esutil/recfile/records.cpp -o build/temp.macosx-10.7-x86_64-3.7/esutil/recfile/records.o -arch i386 -arch x86_64
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the
libc++ standard library instead [-Wstdlibcxx-not-found]
In file included from esutil/recfile/records.cpp:1:
esutil/recfile/records.hpp:2:10: fatal error: 'iostream' file not found

Could someone recommend how to solve this? Thanks

Danny

Is there a way to get the Ra Dec of the trixel centers?

Hi,

I am interested in making a grid on a sphere. It would be really helpful if you could tell me the spherical coordinates (Ra Dec) of the centers of the trixels. An advantage of using htms is that the centers of 1/4th of the trixels are maintained if the grid is made finer. Please reply as soon as possible.
Thanks
Regards
Arvind B

Pip version of esutil has syntax error

Hi esheldon,

I just wanted to let you know that when installing with pip I find that is an error on import. It appears that there is an extra comma here which should be removed after the kwargs:

Traceback (most recent call last):
File "src/models/make_models.py", line 11, in
from esutil.cosmology import Cosmo
File "/u/jacobic/.local/anaconda3/envs/ezgal/lib/python2.7/site-packages/esutil/init.py", line 107, in
from . import integrate
File "/u/jacobic/.local/anaconda3/envs/ezgal/lib/python2.7/site-packages/esutil/integrate/init.py", line 3, in
from . import util
File "/u/jacobic/.local/anaconda3/envs/ezgal/lib/python2.7/site-packages/esutil/integrate/util.py", line 20, in
from .. import stat
File "/u/jacobic/.local/anaconda3/envs/ezgal/lib/python2.7/site-packages/esutil/stat/init.py", line 11, in
from . import util
File "/u/jacobic/.local/anaconda3/envs/ezgal/lib/python2.7/site-packages/esutil/stat/util.py", line 981
sdev=False, **ignored_kw,
^
SyntaxError: invalid syntax

After removing it manually it seems to work

Cheers,
Jacob

Please specify and include license

Hi,

I'd like to package esutil for Fedora, therefore I need information on the licensing of the package. Can you specify the license and add a copy to the source tree? Thanks :)

Greetings,
Christian

small bug/typo in wcsutil.py (too minor for a pull request)

There is a small bug/typo in the file wcsutil.py, line 597:

It currently is:

x,y = self._findxy_one(lon, lat, xtol=tol)

but it must read:

x,y = self._findxy_one(lon, lat, xtol=xtol)

(tol -> xtol at the end of the line)

numpy_util.match should work for strings

let ints and strings go, but not floats

hdfs ls or du

Using hdfs ls or du command i receive this error message:

a bytes-like object is required, not 'str'
Traceback (most recent call last):
File "xxxxxxxxxxx/esutil/hdfs.py", line 139, in du
lines = stdo.split("\n")
TypeError: a bytes-like object is required, not 'str'

Add conda installation to readme

I was having some trouble installing on NERSC until I realized that the code is on conda-forge! Would be helpful to put that in the readme so future people don't make my same mistake.

convert tests to pytest

setup CI

htm package is not yet there

What steps will reproduce the problem?
1. svn checkout http://esutil.googlecode.com/svn/trunk/ esutil-read-only
2. sudo python setup.py install
3. ipython
4. import esutil as es


The htm subpackage is not associated with es but other subpackages are all
there.

Original issue reported on code.google.com by [email protected] on 5 Mar 2010 at 11:31

Matcher segfaults if a dummy HTM object isn't created first

The following code snippet seg-faults (a fresh python instance):

import esutil
import numpy as np
m=esutil.htm.Matcher(10,np.array([10.0]),np.array([10.0]))

But initializing a dummy object first works:

import esutil
import numpy as np
h=esutil.htm.HTM(10)
m=esutil.htm.Matcher(10,np.array([10.0]),np.array([10.0]))

esutil pip install fails on github actions using ubuntu-latest (22.04)

I use pip in my github actions for redmapper. And the esutil installation failed today, apparently caused by the update of ubuntu-latest from 20.04 to 22.04 in November/December (it was a rolling release). Specifically requesting ubuntu-20.04 fixed the issue (temporarily).

The pip install error was the following:

      esutil/htm/htmc.h:111:47: error: ISO C++17 does not allow dynamic exception specifications
        111 |                         const char* filename) throw (const char *);
            |                                               ^~~~~

Which is strange because it should be explicitly setting c++11, but I noticed that the build line looks like:

      gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/numpy/core/include -Iesutil/include -Iesutil/recfile -Iesutil/htm -Iesutil/htm/htm_src -I/opt/hostedtoolcache/Python/3.8.15/x64/include/python3.8 -c esutil/htm/htm_src/SpatialInterface.cpp -o build/temp.linux-x86_64-3.8/esutil/htm/htm_src/SpatialInterface.o

(That is, no use of the -std=c++11 which should be there.)

use great circle arc distance when computing WCS jacobian

Failing to import esutil

debian:~/code/git/esutil$ python3 setup.py install --home=/opt/local/esutil/0.6.5/
running install
running build
running build_py
running build_ext
running install_lib
creating /opt/local/esutil/0.6.5/lib
creating /opt/local/esutil/0.6.5/lib/python
creating /opt/local/esutil/0.6.5/lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/oracle_util.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/sqlite_util.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/random.py -> /opt/local/esutil/0.6.5//lib/python/esutil
creating /opt/local/esutil/0.6.5/lib/python/esutil/unit_tests
copying build/lib.linux-x86_64-3.5/esutil/unit_tests/hist_tests.py -> /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests
copying build/lib.linux-x86_64-3.5/esutil/unit_tests/__init__.py -> /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests
copying build/lib.linux-x86_64-3.5/esutil/unit_tests/int_tests.py -> /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests
copying build/lib.linux-x86_64-3.5/esutil/unit_tests/sfile_tests.py -> /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests
copying build/lib.linux-x86_64-3.5/esutil/unit_tests/pickle_tests.py -> /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests
copying build/lib.linux-x86_64-3.5/esutil/unit_tests/htm_tests.py -> /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests
copying build/lib.linux-x86_64-3.5/esutil/sfile.py -> /opt/local/esutil/0.6.5//lib/python/esutil
creating /opt/local/esutil/0.6.5/lib/python/esutil/recfile
copying build/lib.linux-x86_64-3.5/esutil/recfile/Util.py -> /opt/local/esutil/0.6.5//lib/python/esutil/recfile
copying build/lib.linux-x86_64-3.5/esutil/recfile/test.py -> /opt/local/esutil/0.6.5//lib/python/esutil/recfile
copying build/lib.linux-x86_64-3.5/esutil/recfile/_records.cpython-35m-x86_64-linux-gnu.so -> /opt/local/esutil/0.6.5//lib/python/esutil/recfile
copying build/lib.linux-x86_64-3.5/esutil/recfile/__init__.py -> /opt/local/esutil/0.6.5//lib/python/esutil/recfile
copying build/lib.linux-x86_64-3.5/esutil/recfile/records.py -> /opt/local/esutil/0.6.5//lib/python/esutil/recfile
copying build/lib.linux-x86_64-3.5/esutil/__init__.py -> /opt/local/esutil/0.6.5//lib/python/esutil
creating /opt/local/esutil/0.6.5/lib/python/esutil/stat
copying build/lib.linux-x86_64-3.5/esutil/stat/__init__.py -> /opt/local/esutil/0.6.5//lib/python/esutil/stat
copying build/lib.linux-x86_64-3.5/esutil/stat/unit_tests.py -> /opt/local/esutil/0.6.5//lib/python/esutil/stat
copying build/lib.linux-x86_64-3.5/esutil/stat/_stat_util.cpython-35m-x86_64-linux-gnu.so -> /opt/local/esutil/0.6.5//lib/python/esutil/stat
copying build/lib.linux-x86_64-3.5/esutil/stat/util.py -> /opt/local/esutil/0.6.5//lib/python/esutil/stat
copying build/lib.linux-x86_64-3.5/esutil/stat/_chist.cpython-35m-x86_64-linux-gnu.so -> /opt/local/esutil/0.6.5//lib/python/esutil/stat
copying build/lib.linux-x86_64-3.5/esutil/ostools.py -> /opt/local/esutil/0.6.5//lib/python/esutil
creating /opt/local/esutil/0.6.5/lib/python/esutil/cosmology
copying build/lib.linux-x86_64-3.5/esutil/cosmology/cosmology.py -> /opt/local/esutil/0.6.5//lib/python/esutil/cosmology
copying build/lib.linux-x86_64-3.5/esutil/cosmology/__init__.py -> /opt/local/esutil/0.6.5//lib/python/esutil/cosmology
copying build/lib.linux-x86_64-3.5/esutil/cosmology/_cosmolib.cpython-35m-x86_64-linux-gnu.so -> /opt/local/esutil/0.6.5//lib/python/esutil/cosmology
copying build/lib.linux-x86_64-3.5/esutil/hdfs.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/fits.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/coords.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/pbar.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/misc.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/algorithm.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/xmltools.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/wcsutil.py -> /opt/local/esutil/0.6.5//lib/python/esutil
creating /opt/local/esutil/0.6.5/lib/python/esutil/integrate
copying build/lib.linux-x86_64-3.5/esutil/integrate/__init__.py -> /opt/local/esutil/0.6.5//lib/python/esutil/integrate
copying build/lib.linux-x86_64-3.5/esutil/integrate/_cgauleg.cpython-35m-x86_64-linux-gnu.so -> /opt/local/esutil/0.6.5//lib/python/esutil/integrate
copying build/lib.linux-x86_64-3.5/esutil/integrate/util.py -> /opt/local/esutil/0.6.5//lib/python/esutil/integrate
copying build/lib.linux-x86_64-3.5/esutil/cosmology_purepy.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/numpy_util.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/stomp_util.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/plotting.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/json_util.py -> /opt/local/esutil/0.6.5//lib/python/esutil
copying build/lib.linux-x86_64-3.5/esutil/io.py -> /opt/local/esutil/0.6.5//lib/python/esutil
creating /opt/local/esutil/0.6.5/lib/python/esutil/htm
copying build/lib.linux-x86_64-3.5/esutil/htm/__init__.py -> /opt/local/esutil/0.6.5//lib/python/esutil/htm
copying build/lib.linux-x86_64-3.5/esutil/htm/unit_tests.py -> /opt/local/esutil/0.6.5//lib/python/esutil/htm
copying build/lib.linux-x86_64-3.5/esutil/htm/_htmc.cpython-35m-x86_64-linux-gnu.so -> /opt/local/esutil/0.6.5//lib/python/esutil/htm
copying build/lib.linux-x86_64-3.5/esutil/htm/htm.py -> /opt/local/esutil/0.6.5//lib/python/esutil/htm
copying build/lib.linux-x86_64-3.5/esutil/htm/htmc.py -> /opt/local/esutil/0.6.5//lib/python/esutil/htm
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/oracle_util.py to oracle_util.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/sqlite_util.py to sqlite_util.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/random.py to random.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests/hist_tests.py to hist_tests.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests/__init__.py to __init__.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests/int_tests.py to int_tests.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests/sfile_tests.py to sfile_tests.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests/pickle_tests.py to pickle_tests.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/unit_tests/htm_tests.py to htm_tests.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/sfile.py to sfile.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/recfile/Util.py to Util.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/recfile/test.py to test.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/recfile/__init__.py to __init__.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/recfile/records.py to records.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/__init__.py to __init__.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/stat/__init__.py to __init__.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/stat/unit_tests.py to unit_tests.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/stat/util.py to util.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/ostools.py to ostools.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/cosmology/cosmology.py to cosmology.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/cosmology/__init__.py to __init__.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/hdfs.py to hdfs.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/fits.py to fits.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/coords.py to coords.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/pbar.py to pbar.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/misc.py to misc.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/algorithm.py to algorithm.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/xmltools.py to xmltools.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/wcsutil.py to wcsutil.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/integrate/__init__.py to __init__.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/integrate/util.py to util.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/cosmology_purepy.py to cosmology_purepy.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/numpy_util.py to numpy_util.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/stomp_util.py to stomp_util.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/plotting.py to plotting.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/json_util.py to json_util.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/io.py to io.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/htm/__init__.py to __init__.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/htm/unit_tests.py to unit_tests.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/htm/htm.py to htm.cpython-35.pyc
byte-compiling /opt/local/esutil/0.6.5//lib/python/esutil/htm/htmc.py to htmc.cpython-35.pyc
running install_egg_info
Writing /opt/local/esutil/0.6.5//lib/python/esutil-0.6.5.egg-info

debian:~/code/git/esutil$ python3
Python 3.5.3 (default, Nov 18 2020, 21:09:16) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import esutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/npkd46/code/git/esutil/esutil/__init__.py", line 113, in <module>
    from . import integrate
  File "/home/npkd46/code/git/esutil/esutil/integrate/__init__.py", line 1, in <module>
    from . import util
  File "/home/npkd46/code/git/esutil/esutil/integrate/util.py", line 20, in <module>
    from .. import stat
  File "/home/npkd46/code/git/esutil/esutil/stat/__init__.py", line 27, in <module>
    from . import util
  File "/home/npkd46/code/git/esutil/esutil/stat/util.py", line 69, in <module>
    from . import _stat_util
ImportError: cannot import name '_stat_util'
>>> 

Any ideas?

missing terminating " character in esutil/recfile/records.cpp

What steps will reproduce the problem?
1. python setup.py build

What is the expected output? What do you see instead?

No error should be reported. Instead, I get:

In file included from esutil/recfile/records.cpp:1:0:
esutil/recfile/records.hpp:33:1: warning: missing terminating " character
esutil/recfile/records.hpp:90:1: warning: missing terminating " character
esutil/recfile/records.hpp:105:3: warning: missing terminating " character
esutil/recfile/records.hpp:132:56: warning: missing terminating " character
esutil/recfile/records.hpp:142:3: warning: missing terminating " character
esutil/recfile/records.hpp:169:3: warning: missing terminating " character
esutil/recfile/records.hpp:178:3: warning: missing terminating " character
esutil/recfile/records.hpp:182:3: warning: missing terminating " character

However, the code compiles.

Original issue reported on code.google.com by [email protected] on 29 Dec 2010 at 8:08

Cannot import esutil

Hi there, I successfully installed esutil, and when I tried to import the package, it gives me this error:
ImportError: /home/tao/anaconda2/lib/python2.7/site-packages/esutil/recfile/_records.so: undefined symbol: _ZTVNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE

Any idea how to fix this? Thanks!

esutil assumes that on Darwin Python was built with both `i386` and `x86_64`

esutil assumes that on Darwin (Mac OS X) Python was built with both i386 and x86_64 but this isn't necessarily true.

The setup.py file has the following

if platform.system()=='Darwin':
    extra_compile_args=['-arch','i386','-arch','x86_64']
    extra_link_args=['-arch','i386','-arch','x86_64']
else:
    extra_compile_args=[]
    extra_link_args=[]

But, e.g., on my system, Python was only installed with x86_64 libraries.

init_numpy() should explicitly return NULL

There are a few definitions of "static int *init_numpy(void)" in the source tree that call numpy's import_array(). Their definitions should be

static int *init_numpy(void) {
    import_array();
    return NULL;
}

import_array() is a macro that returns if and only if it fails. With the current definition of init_numpy(), the control flow comes out of import_array() if succeess, and returns from init_numpy() without a return statement.

The optimizer of g++ (but not gcc) in gcc-8 assumes that the control flow never reaches a point where the control flow would have to return without a return value. This means that gcc-8 optimizes init_numpy() such that import_array() will always fail, because the control flow would return without a return statement if import_array() were to succeed.

In fact, if I compile the current esutil with gcc-8, esutil.htm.HTM() fails with this error:

RuntimeError: FATAL: module compiled as little endian, but detected different endianness at runtime
ImportError: numpy.core.multiarray failed to import

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/site-packages/esutil/htm/htmc.py", line 152, in __init__
    this = _htmc.new_HTMC(depth)
SystemError: <built-in function new_HTMC> returned a result with an error set

Going beyond the current limitation on HTM

What steps will reproduce the problem?
1. The depth limit of 13 is purely because of the use of 32 bit integers.
2. The underlying C code, if modified to use a long int or a 64 bit will solve 
the problem.
3. The rest of the package many not require specific change.

What is the expected output? What do you see instead?
At depth 14 or beyond, one starts getting negative HTM IDs, this is due to 
overflow of the 32 bit limit on the integer used for computing the same.

What version of the product are you using? On what operating system?
0.5.X, Ubuntu 12.04 LTS, Python 2.7.3

Please provide any additional information below.
None.

Original issue reported on code.google.com by [email protected] on 24 Sep 2014 at 5:58

HTM.match sometimes returns non-zero distance for self-match

I was trying to use HTM.match to get the distance to the nearest neighbour for each object in a catalog. Hence I thought I could use maxmatch=2 and use the non-zero distance. However, with maxmatch=1, HTM.match returns some (about 20%) non-zero distances which presumably should not happen, since every objects' closest object should be itself. I've copied some ipython showing this below:

In [4]: from esutil.htm import HTM

In [5]: h=HTM()

In [7]: ra
Out[7]: 
array([52.18053786, 52.45002301, 52.15060583, ..., 52.39342204,
       52.39235128, 52.41923532])

In [8]: dec
Out[8]: 
array([-27.12791705, -27.12528608, -27.11948308, ..., -27.2711784 ,
       -27.27120412, -27.27125424])

In [9]: m1,m2,d12 = h.match(ra, dec, ra, dec, 0.01, maxmatch=1)

In [10]: d12
Out[10]: 
array([8.53773646e-07, 0.00000000e+00, 0.00000000e+00, ...,
       0.00000000e+00, 8.53773646e-07, 0.00000000e+00])

In [11]: (d12<1.e-9).sum()
Out[11]: 12019

In [12]: (d12>1.e-9).sum()
Out[12]: 2408

In [13]: m1
Out[13]: array([    0,     1,     2, ..., 14424, 14425, 14426])

Reverse indices returned by histogram2d do not respect xmin, xmax, ymin, and ymax keywords

Hello,

It seems reverse indices returned by histogram2d are returned with respect to 
the entire x and y arrays supplied to histogram2d, even when xmin, xmax, ymin, 
and ymax are used. I would expect histogram2d to return indices only for 
elements in ranges specified by xmin, xmax, ymin, and ymax keywords.

Is this a bug?

Cheers,

Brani

Original issue reported on code.google.com by [email protected] on 4 Jun 2011 at 11:06

Installation issues in Ubuntu

I'm running 64-bit Ubuntu 15.04. Installing esutil fails with both:
sudo pip3 install esutil
sudo python3 setup.py (in the appropriate folder)

I can install other packages and my python setup seems otherwise well behaved. I have python3-dev installed.

Here's a terminal dump from the pip install: http://pastebin.com/eXbAEqcG . I've spent a long time trying to parse it but I can't seem to find any 'fatal' errors. Any ideas what the cause is?

htm center positions

Perhaps an HTM n00b question, I see I can input an RA,Dec pair and get the HTM id for it, is there an easy way to get the RA,Decs of all the trixel centers for a given depth?

add function to numpy_util to combine fields from two arrays

Make an array from the combined set of fields

a1 = np.zeros(n, dtype=[('ra', 'f8'), ('dec', 'f8')])
a2 = np.zeros(n, dtype=[('id', 'i8'), ('name', 'U10')])

# this makes a new array with the combined set of fields
a = eu.numpy_util.combine_fields(a1, a2)

other possible names

a = eu.numpy_util.combine_arrays(a1, a2)
a = eu.numpy_util.combine_array_fields(a1, a2)

fails to compile and install with the compiling flag -std=c++17

When I try to install esutil in my computer using pip, The install process reports errors. The reason seems like that my compiler default C++ standard is C++17, and the c++ code includes throw exception flags that are only supported in C++11. Maybe you can add a -std=c++11 flag when compiling, or delete the exception statement in the header file. The failure report is below:

esutil/htm/htmc.h:29:36: error：ISO C++17 does not allow dynamic exception specifications 29 | HTMC(int depth=10) throw (const char *); | ^~~~~ esutil/htm/htmc.h:30:33: error：ISO C++17 does not allow dynamic exception specifications 30 | void init(int depth=10) throw (const char *); | ^~~~~ esutil/htm/htmc.h:39:11: error：ISO C++17 does not allow dynamic exception specifications 39 | ) throw (const char *); | ^~~~~ esutil/htm/htmc.h:46:30: error：ISO C++17 does not allow dynamic exception specifications 46 | ) throw (const char *); | ^~~~~ esutil/htm/htmc.h:78:31: error：ISO C++17 does not allow dynamic exception specifications 78 | throw (const char *); | ^~~~~ esutil/htm/htmc.h:100:32: error：ISO C++17 does not allow dynamic exception specifications 100 | PyObject* dec) throw (const char *);

Installation from pip on a clean system is broken

When installing on a clean system using pip install esutil the installation goes fine, but the package cannot be imported.
The error is:

root@836b9e6d53c4:/# python3 -c 'import esutil'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/esutil/__init__.py", line 104, in <module>
    from . import integrate
ImportError: cannot import name 'integrate' from partially initialized module 'esutil' (most likely due to a circular import) (/usr/local/lib/python3.10/dist-packages/esutil/__init__.py)

Steps to replicate (using Ubuntu docker image as an example, just to have a clean system - but it manifests on other distributions as well):

docker run -it ubuntu
> apt update
> apt install python3-pip
> pip3 install esutil
> python3 -c 'import esutil'

stat.histogram runs Binner.calc_stats() twice

The call to stat.histogram calls Binner.do_hist() which defaults calc_stats=True. Then, histogram() calls Binner.calc_stats() again if necessary (weighted histogram, or more stats).

This leads to a performance degradation (30% slower in one particular use case).

Solution would be to change the call in stat.histogram() to:
b.do_hist(...,calc_stats=False)